The tools you will need to protect an open online community from abuse:
- Abuse Detection
- Manual & automated monitoring
- Automated tools must be able to adapt to new forms of abuse as they emerge
- User-visible “Report abusive content” feature
- Fills gaps in internal content monitoring
- Helps teach you what your specific community considers “abusive.”
- Review queue for internal team
- Collect & prioritize abuse reports
- Suggest appropriate resolutions based on history of user’s past behavior
- Must be as quick/easy as possible, since it will be scanned frequently
- (maybe) Provide feedback to users when reports are resolved. Be careful, this can lead to frustration if user disagrees with your judgment.
- Muting tools
- Time-limited and permanent mute (read-only mode)
- Provides “cooling-off” time without permanent harm
- Banning tools
- Time-limited and permanent bans
- (maybe) “Shadowbans” to slow down adversary response
- Reputation Database
- Prevent abusers from returning under different names/accounts
- Track: IP addresses, email addresses, VPNs, social network accounts, browser/device fingerprints
- Anti-Fraud Firewall
- Close off channels that abusers use to target the community
- Anonymizing Proxies/VPNs, Throw-away email providers, Datacenters, Country-level blocks, rate limits
- Identity verification to guard posting privileges
- e.g., social network login or SMS phone line
- Note: do not rely on Google or Facebook OAuth alone to authenticate identity. They are bad at this.
- Honeypot / “Lightning Rod”
- Divert troublemakers to a well-confined area
- Pro-active detection & response
- Look for signs of incoming abuse before it happens
- Deflect in a positive direction, or pre-emptively mute
Nice-to-have improvements: