Before collecting public social media data for a research project, the decision you face is not which tool to use but whether you can answer seven questions: what the data is for, which access route the platform offers, what the platform's terms allow, whose data you will hold, whether an ethics review is needed, how you will store and delete the data, and how you will publish without exposing anyone. A project that answers all seven before the first request is far less likely to stall halfway, lose its data access or face an objection from an ethics board or a journal. The questions apply to academic studies, nonprofit research and market research alike, although the formal requirements differ.
1. What exactly does the research question need?
Start from the question, not from what the platform makes available. A study of how a public health message spread among news outlets needs posts from a defined list of public organizational accounts over a defined period. It does not need follower lists, profile photos or the comments of private individuals.
Write a one-paragraph data specification that lists the accounts or search terms, the date range, the fields and the reason each field is needed. Compare two hypothetical plans: one collects every post mentioning a keyword with full author profiles; the other collects only post text, timestamps and engagement counts from 200 named public organizational accounts. The second is easier to justify, easier to store safely and usually enough to answer the question. Every field you leave out is a field you never have to protect.
2. Does the platform offer an official API or a research access program?
Most large platforms offer some form of official access: a public API, a paid API tier, or a program for academic and public-interest researchers. Some regulatory regimes also create routes for researcher access to platform data, with conditions set by the law and the platform. What is available, at what price and with which eligibility rules changes often, so check the platform's developer and research pages at the start of the project rather than relying on what was true for an earlier study.
Compare the options against the same criteria: which fields are available, how far back in time the data goes, the volume limits, the cost, how long approval takes and what the agreement requires in return. If an official route provides what the question needs, it is usually the most stable and defensible choice. Collecting from public pages is a fallback for what official routes do not cover, and only where the terms allow it.
Build the approval time into the schedule. Research programs often ask for a project description, an institutional affiliation and a data management plan, and a review can take weeks. A study planned for a single semester can lose most of its collection window waiting for access if the application goes in late. Apply as soon as the research question is fixed, and keep a written record of what the access agreement permits, since later questions from reviewers will often turn on it.
3. What do the platform's terms say about collection and sharing?
Read the platform's terms of service and, if you use an API, its developer terms. Look for three things: whether automated collection outside the API is permitted, how long you may store data, and whether you may share the dataset with other researchers.
The sharing rules often surprise researchers at the end of a project. Some platforms' terms have limited sharing to post identifiers rather than full content, so that others must retrieve the posts themselves, and posts deleted in the meantime are no longer available. If a journal or funder expects you to publish your dataset, find out before collection whether the platform's terms allow it, and plan the data availability statement accordingly.
4. Whose data will end up in the dataset?
Public social media data is still data about people. Even a project focused on organizational accounts will collect replies, mentions and quoted posts from individuals. Under privacy laws such as the GDPR, information about identifiable people is personal data whether or not it is publicly visible, and some categories, such as health, political opinions, religion or sexual orientation, receive extra protection.
Ask three things about the people in your data: are they public figures speaking in a public role, or private individuals; could the topic expose them to harm if they were identified; and could any of them be minors. A study of corporate announcements and a study of posts about a mental health condition raise very different questions, even if the collection method is identical. The higher the sensitivity, the stronger the case for aggregation, minimization and formal review. A lawyer or the institution's data protection officer should review a project that involves sensitive categories.
5. Does the project need an ethics review?
In universities and many research organizations, work involving data about people usually goes through an ethics committee or, in the United States, an institutional review board. Whether a study of public posts counts as human subjects research, and whether it qualifies for an exemption, depends on the institution's policies and the design of the study. Do not assume an exemption because the data is public; ask the board early and let it make that determination.
Outside academia there may be no formal board, but the same questions apply. A short internal review, with someone outside the project reading the data specification and the answers to questions 4, 6 and 7, catches most problems before they become expensive.
6. How will the data be stored, secured, kept and deleted?
Write a data management plan before collection, not after. It should say where the data will be stored, who can access it, how access is logged, how long the raw data will be kept and what happens to it at the end of the project.
Two points deserve particular attention. First, separate identifiers from content where you can: store user identifiers in a protected table and work with pseudonymous keys in the analysis files. Second, decide how you will handle posts that are later deleted or accounts that become private. A common approach is to remove them from the working dataset at regular intervals, which respects the person's choice and is often what platform terms require. A hypothetical study holding 50,000 posts for two years might re-check availability every quarter and remove what has gone.
Security deserves the same planning as the analysis. Keep the raw data off personal laptops and shared drives without access control, encrypt storage where the institution provides it, and limit access to the people named in the plan. When the project ends, delete the raw data on the date the plan set, or archive it under the conditions the ethics review approved, and record that you did so.
7. How will you publish results without exposing individuals?
Publication is where public data most often causes harm. Quoting a post word for word in a paper or report can let any reader find the original with a search engine, and with it the person who wrote it. That risk is highest for private individuals and sensitive topics.
Practical safeguards include reporting results in aggregate, paraphrasing quotations so they cannot be searched back to the source, removing usernames and identifying details from examples, and asking permission when a verbatim quote is essential. Public figures speaking in their official role are a different case, and quoting them is usually less of a concern. Decide these rules during planning, so they shape what you collect.
Check the paraphrase test before submitting. Take each quotation in the draft, search for a distinctive phrase from it, and see whether the original post turns up. If it does, rewrite the quotation further or replace it with a description of the pattern it illustrates. Images and screenshots of posts carry the same risk and usually more, because they show usernames and profile pictures along with the text.
The same balance between usefulness and respect for the people in the data runs through the advice on mining social media data legally and effectively and in the myths about collecting public personal data under the GDPR and CCPA. If part of the collection is handed to a managed provider such as ScrapingIsNotACrime, give the provider the written answers to these seven questions as part of the specification, so that the access route, the fields and the retention rules are the ones your review approved.
