The CDO Council is excited to share the results of a pilot project that explored using state-of-the-art Natural Language Processing (NLP) techniques to aid and expedite comment analysis in the federal rulemaking process.
PROJECT BACKGROUND
When a Congressional bill becomes law, agencies may need to draft regulations and publish them to implement those laws. The regulatory review process provides the public with an opportunity to participate in the rulemaking process. The federal government publishes tens of thousands of documents each year in the Federal Register, which garner millions of comments from the public – and each agency is responsible for reviewing and responding to the comments submitted on their regulations.
To meet their legal obligation to review and incorporate all relevant comments, many bureaus perform very sophisticated and tailored analyses to understand this public response. These analyses require significant dedication of staff time and resources and many of the questions are common and could even be solved in a generalizable way across government. For example, how many of these comments were submitted as part of a mass mail campaign? What topics are the public concerned with in this proposed regulation? How can I quickly review and share these comments with my organization’s subject matter experts for review and response?
THE OPPORTUNITY
The CDO Council team recognized an opportunity to leverage recent advances in NLP technology to create high-performing, standardized tools that could be used in any agency’s comment analysis process. This would allow agencies to gain efficiencies through novel insights and streamlined processing of comments, realize cost savings, and reduce duplicative development efforts across government.
USDA and the CDO Council worked with stakeholder agencies across the government – Environmental Protection Agency (EPA), Federal Communications Commission (FCC), Federal Energy Regulatory Commission (FERC), USDA Forest Service, and USDA Food and Nutrition Service – to identify the comment analysis tools that would provide most value to their process and develop a proof-of-concept.
THE CDO COUNCIL COMMENT ANALYSIS TOOLSET
This pilot successfully developed high-performing tools that use NLP to aid any agency’s comment analysis process. The team leveraged state-of-the-art neural network models based on empirically validated research mostly released within the past two years. These models are trained on massive amounts of text and bring with them knowledge of the English language, as well as world knowledge. These base models have shown a high degree of performance across our participating agencies, regardless of technical content or comment type. This flexibility allows any federal government agency to adopt the “generalizable” models and software to improve their own comment analysis process. The team also identified opportunities to further fine-tune, or “customize,” the models to meet agency-specific missions and needs – and to craft learning models to specific agency language.
We developed tools, applicable to agencies across government, that help comment reviewers:
-
Identify the topics and themes found within comments under a regulation (categorizing)
-
Identify and group comments that are duplicates or near duplicates based on text matching (e.g., form letters or mass mail campaigns)
-
Identify and group comments that are semantically similar meaning they contain similar ideas and can be more efficiently grouped and shared for subject matter expert review and response
The team also developed a clickable prototype that demonstrates how these tools can be integrated into a comment analyst’s current workflow.
BENEFITS TO AGENCIES AND GOVERNMENT-WIDE
These tools offer significant value at the agency level by helping reviewers respond to comments more quickly and easily. They can also offer new and better insights to comment analysts – for example, stakeholders indicated that identifying the meaning or ideas within groups of comments would benefit their review. Cost savings can be realized as agencies gain efficiencies and reduce up-front costs of comment analysis tool development.
From a government-wide perspective, sharing these tools can improve development efforts at individual agencies. Services such as Regulations.gov/FDMS have created many cross-agency efficiencies in the rulemaking space. This pilot suggests the potential to create additional efficiencies and standardize comment analysis tools across the federal government by leveraging the latest in NLP advancements in technology.
WHERE CAN I LEARN MORE?
Additional details on the pilot, tools and prototype developed, and recommendations for government-wide implementation can be found in the CDO Council Comment Analysis Pilot Final Recommendations report, published in July 2021. You can also access the code from this project. Contact Ted.Kaouk@usda.gov with any questions.