Ensuring Data Sovereignty: A Practical Guide for Public Agencies

Ensuring Data Sovereignty: A Practical Guide for Public Agencies

By Christian Ott, eFile Project Lead

Upgrading an agency’s digital infrastructure is a major undertaking for any municipal administration. Whether an agency is launching a new public disclosure platform, changing agenda management systems, or modernizing records workflows, the goal is usually the same: reduce manual work, improve transparency, and elevate the user experience for both staff and the public.

A major operational risk is often overlooked during the procurement process. When evaluating a new platform, agencies naturally focus on onboarding timelines and modern features. However, the immediate bottleneck to launching that new software is the initial data migration. The true technical challenge lies in extracting complete, usable historical records from your incumbent vendor so your new system can actually get started. If the outgoing provider delivers degraded or incomplete data, the transition stalls before it even begins.

Most vendor agreements contain language stating that the public agency owns its data. That language matters, but legal ownership alone is not enough. If an agency cannot retrieve its records in a complete, organized, and usable format, then ownership becomes difficult to exercise in practice.

For city clerks and other municipal administrators, this is not simply a technology concern. It is a public records concern. Agencies need to know that their historical records can be searched, verified, migrated, preserved, and made available when needed. That requires more than a vendor handing over a collection of files at the end of a contract.

It requires data sovereignty in a practical sense: the ability of the public agency to retain meaningful control over its own records throughout the entire life of the system, including when the vendor relationship ends.

Where Standard Vendor Exports Often Fall Short

The primary challenge during a system transition is rarely a flat refusal by a vendor to provide any data. More often, the problem is that the export is incomplete, flattened, or disconnected from the structure that made the data useful inside the original system.

A vendor may provide spreadsheets, reports, or folders of documents that appear complete at first glance. But if the export does not preserve the relationships between records, the unique identifiers that distinguish one record from another, or the metadata needed to understand each file, the agency may be left with information that is technically present but operationally difficult to use.

Common problems include:

1. Missing Unique Identifiers

An agency may receive a spreadsheet listing filers, committees, documents, or filings by name, but without the system-generated IDs that uniquely identify each record.

This creates problems when:

  • Two people have the same or similar names.
  • A filer changes name, position, or role over time.
  • A committee reorganizes or changes members.
  • Multiple records exist for the same person.
  • Historical documents and attachments lose their relational links.

Names are useful for people, but they are not enough for accurate data migration. Stable system identifiers are what allow records to remain distinct and relationships stay discoverable over many years.

2. Broken Relationships Between Records

Many public agency systems store information in connected pieces. A filer may be connected to one or more filings. A filing may be connected to uploaded documents, timestamps, amendments, review notes, audit history, or public access records. A committee may be connected to transactions, officers, forms, and reporting periods.

If a vendor provides those pieces as separate files without preserving the links between them, the agency may have the raw information but not the structure needed to understand how the records fit together.

When relationships are broken, staff or the incoming vendor may have to reconstruct the record history manually. That process is time-consuming, expensive, and prone to error.

3. Unindexed Document Dumps

Another common issue occurs when an agency receives a large folder of PDFs, scans, or uploaded files without a clear index explaining what each document is and linking it to other records.

A folder containing thousands of files is not, by itself, a complete document export. The agency also needs a machine-readable index that connects each file to the correct record.

Without this kind of index, documents may become detached from the records they were meant to support. That can make it harder to respond to public records requests, verify historical filings, or migrate records into a replacement system.

The Technical Pillars of Data Completeness

To protect public records during a software transition, agencies should evaluate vendor exports against several practical requirements. These requirements may sound technical, but the purpose is straightforward: the agency should be able to retrieve its records in a form that remains complete, understandable, and usable outside the vendor’s system.

1. Relational Database Integrity

Many systems are built around relationships between different types of records. For example, a filing may be connected to a filer, a filing period, electronic transactions or filing records, one or more documents, and a history of submissions or amendments. These relationships are what allow the system to present a complete and accurate record.

A complete export should preserve that structure. The data should not be flattened into isolated spreadsheets if doing so removes the connections between records.

Acceptable export formats may include:

  • SQL database exports, such as MS SQL, PostgreSQL or MySQL dumps.
  • Structured CSV files that preserve primary keys and foreign keys.
  • JSON exports that clearly define related records and identifiers.
  • Document manifests that connect uploaded files to database records.

The specific format may vary depending on the system, but the principle should not: the export must preserve the relationships needed to reconstruct the agency’s records accurately.

2. Granular Uniqueness

Every important record in the system should have a stable, system-generated identifier. This includes not only major entities, such as filers or committees, but also individual filings, documents, transactions, amendments, and other related records.

These identifiers are important because they prevent accidental record merging or misclassification during migration. They also allow the incoming system or technical team to distinguish between records that may look similar to a human reviewer but are distinct in the database.

Without identifiers, the agency may be forced to rely on names, dates, or file labels alone. That is not a reliable foundation for long-term records management.

3. Integrated Metadata and Document Manifests

Public records systems often include unstructured files, such as PDFs, scans, attachments, and uploaded forms. These documents must remain connected to the structured data that explains what they are.

A complete export should therefore include a document manifest or metadata index. This index should map each exported file to the correct database record and provide enough information for staff, auditors, or an incoming vendor to understand the document’s context.

A good manifest should answer basic questions:

  • What is this file?
  • Who submitted it?
  • Which record does it belong to?
  • When was it received?
  • What filing period or form type does it relate to?
  • Has it been amended, reviewed, superseded, or replaced?

This information is essential for maintaining the integrity of historical records after the original system is no longer in use.

Contractual Recommendations for Procurement

Public agencies can reduce the risk of data access problems by addressing offboarding requirements before a contract is signed. Data export should not be treated as an afterthought or as an optional service to be negotiated only when the agency is already trying to leave.

Procurement documents, RFPs, and vendor agreements should define what a complete data handoff looks like. The goal is to make expectations clear from the beginning and to ensure that the agency retains practical control over its own records.

Agencies should consider including the following requirements.

1. No-Fee Data Offboarding

The vendor should provide complete data and document exports at the end of the contract without charging additional offboarding fees.

If the records belong to the public agency, the agency should not have to pay a penalty to retrieve them in a usable format. Any costs, timelines, or procedures related to export should be clearly addressed in the agreement.

2. Open, Machine-Readable Formats

Exports should be delivered in open, machine-readable formats that can be reviewed, validated, and imported into another system.

Depending on the system, this may include:

  • SQL database dumps.
  • Structured CSV files.
  • JSON exports.
  • Document manifests.
  • Metadata indexes.
  • File archives with clearly mapped filenames.

Flat reports or screenshots are not enough. The export must contain the underlying structure needed to preserve the agency’s records.

3. Preservation of System Identifiers and Relationships

The agreement should require the vendor to preserve all primary keys, foreign keys, system identifiers, and relationship mappings.

In plain language, this means the export should not merely list records. It should show how those records are connected. These relationships are essential for accurate migration and long-term record continuity.

4. Complete Document Indexing

If the system stores PDFs, scans, forms, or attachments, the vendor should provide a document index that connects each file to the correct record.

The index should be delivered in a format that can be used by staff or by an incoming vendor, not merely as an informal description. Without this index, document exports can quickly become difficult to search, verify, or restore.

5. Defined Turnaround Times

The contract should specify how quickly the vendor must provide a complete export after a formal request.

This is especially important when a transition is already underway. Delays in export delivery can create operational gaps, delay implementation, and increase the risk that records will be missed or mishandled.

The Importance of a Two-Phase Migration Process

A successful system transition often requires more than a single export. In many cases, agencies need a phased process that allows the incoming vendor and the agency to test and validate the migration before the legacy system is turned off.

This is particularly important for public-facing filing and compliance systems, where activity may continue during the transition period. Agencies may not be able to freeze filings, uploads, amendments, or staff review activity for an extended period while vendors coordinate behind the scenes.

For that reason, contracts should account for both an initial staging export and a final synchronization.

Phase 1: Initial Staging Export

The initial export should be provided while the legacy system is still active. This allows the incoming vendor and/or the agency’s implementation team to load the data into a staging or test environment.

The purpose of this first export is to:

  • Map fields between the old and new systems.
  • Test the import process.
  • Identify missing data or formatting issues.
  • Validate record counts and document links.
  • Confirm that historical records appear correctly.
  • Give staff an opportunity to review sample migrated records before go-live.

This step reduces risk, because problems can be identified before the agency is relying on the new system for daily operations.

Phase 2: Final Delta Synchronization

After the legacy system is formally decommissioned, the outgoing vendor should provide a final export capturing any changes that occurred after the initial staging export.

This final synchronization should include the “delta,” meaning the records that were added, changed, amended, uploaded, or reviewed during the transition window.

This final export should be delivered promptly so the incoming system can be brought fully up to date without a prolonged gap in records access.

Conclusion: Data Ownership Must Include Practical Control

For public agencies, data sovereignty should mean more than a sentence in a contract. It should mean that the agency can retrieve, understand, verify, migrate, and preserve its records throughout the full life of the software relationship.

That requires planning before procurement is complete. Agencies should ask vendors how exports are handled, whether unique identifiers are preserved, whether documents are indexed, whether relationships between records remain intact, and whether the vendor supports both initial exports and final synchronization.

These questions may sound technical, but they are ultimately about stewardship of public records. Municipal clerks and public administrators are responsible for records that must remain accurate, accessible, and trustworthy over time. The systems used to manage those records should strengthen that responsibility, not make it harder to fulfill.

A modern software platform should improve transparency, reduce administrative burden, and support long-term continuity. It should never leave a public agency struggling to regain meaningful access to its own records.