Data Cleaning¶

The data cleaning module helps remove entries based on abbreviations, periodic elements and resolve arithmetic expressions, fractional compositions, etc. along with bracket standardization in extracted chemical formulas. It also includes advanced composition cleaning features to transform raw compositions into standardized, resolved forms.

Basic Usage¶

from comproscanner import ComProScanner

# Initialize scanner
scanner = ComProScanner(main_property_keyword="piezoelectric")

# Clean extracted data
scanner.clean_data(
    json_results_file="extracted_results.json"
)

Parameters¶

Required Parameters¶

json_results_file (str)¶

Path to the JSON results file containing extracted data that needs to be cleaned.

Optional Parameters¶

is_save_separate_results (bool)¶

Whether to save separate cleaned results files.

cleaned_json_results_file (str)¶

Path to the cleaned JSON results file with articles having relevant composition-property data.

is_save_composition_property_file (bool)¶

Whether to save composition-property values to a separate file as a dictionary.

composition_property_file (str)¶

Path to the cleaned composition-property file containing a dictionary of composition-property data.

cleaning_steps (Union[str, List[str]])¶

Either the string "all" (default, every optional step enabled) or a list of step names selecting exactly which optional steps run:

• abbreviation_filtering: drops composition keys containing 2+ consecutive capital letters (abbreviations/junk keys, e.g. "PVDF").
• element_validation_strict: keep only compositions whose key fully resolves to valid periodic-table element symbols.
• element_validation_lenient: a weaker companion of element_validation_strict. Instead of requiring the entire key to be pure elements, it keeps a composition as long as it contains at least one embedded formula fragment anywhere in the text. For example, "Cellulose nanofibers/BaTiO3@TiO2/Polyvinylidene fluoride-(%)" is kept because "BaTiO3" and "TiO2" each parse as elements, even though the rest is descriptive text. Only compositions with no recognizable formula fragment anywhere are dropped. If element_validation_strict is also selected, its stricter result wins: selecting both together gives no additional compositions beyond what element_validation_strict alone would keep.
• text_normalization: deterministic cleanup, strips leading/trailing whitespace, collapses runs of multiple spaces down to one, and title-cases descriptive word tokens (e.g. " Bi4Ti3O12 ultrathin with oxygen vacancies " → "Bi4Ti3O12 Ultrathin with Oxygen Vacancies"). Tokens containing digits (real formula segments) and tokens that fully parse as element symbols (e.g. "NaCl") are left untouched, as are all-caps abbreviations (e.g. "PVDF").
• miller_indices: drops compositions carrying a crystal-plane notation like (002), (111), (100), etc. entirely, rather than stripping the notation and keeping the bare formula. Stripping and keeping would collapse distinct surface-orientation entries for the same material onto the same dict key, e.g. "AlN (002)" and "AlN (110)" would both become "AlN", silently overwriting one value with the other when merged.
• coefficient_expansion_strict: expands leading/trailing/nested bracket coefficient patterns; internally also normalizes trailing zeros and removes zero-coefficient elements as part of expansion. Compositions left with any residual (), [], or * afterward are dropped as unresolved.
• coefficient_expansion_lenient: a weaker companion of coefficient_expansion_strict. Performs the same bracket expansion, but spares compositions with balanced brackets (equal open/close counts, no stray *) whose bracket content is genuine text rather than a failed arithmetic expression, e.g. "(Bi0.5Ag0.5)ZrO3-(as-sintered)" is kept as "Bi0.5Ag0.5ZrO3-(as-sintered)" instead of being dropped as unresolved. If coefficient_expansion_strict is also selected, its stricter unresolved-filtering wins: selecting both together reverts to strict behavior.

Pass an empty list (cleaning_steps=[]) to skip all the optional steps.

is_store_unresolved_compositions (bool)¶

When True, logs a split statistics line showing source, filtered, unresolved, and resolved composition-property pair counts, and saves both filtered compositions (dropped by abbreviation_filtering, element_validation_strict, element_validation_lenient, or miller_indices) and unresolved compositions (still containing parentheses, brackets, or multiplication operators after cleaning) to a JSON file keyed by DOI. Requires is_save_composition_property_file=True.

unresolved_compositions_file (str)¶

Path to the JSON file where filtered and unresolved composition keys are saved, with "filtered" and "unresolved" as top-level keys and DOIs as sub-keys mapping to lists of {"composition": ..., "reason": ...} entries. reason names the step that dropped the composition (e.g. "element_validation_strict", "miller_indices", "unresolved_brackets_or_operators"). Used only when is_store_unresolved_compositions=True.

Cleaning Process Flow¶

The data cleaning process follows this workflow:

graph TD
    A[Start: Raw Extracted Data] --> B{abbreviation_filtering?}
    B -->|selected| B1[Abbreviation Filtering]
    B -->|skipped| C
    B1 --> C{element_validation_strict?}
    C -->|selected| C1[Element Validation - strict]
    C -->|skipped| C2
    C1 --> C2{element_validation_lenient?}
    C2 -->|selected| C3[Element Validation - lenient]
    C2 -->|skipped| D
    C3 --> D{text_normalization?}
    D -->|selected| D1[Text Normalization]
    D -->|skipped| M
    D1 --> M{miller_indices?}
    M -->|selected| M1[Drop compositions with Miller indices]
    M -->|skipped| F
    M1 --> F["Unicode Conversion (always)"]
    F --> G["Arithmetic/Fraction Resolution (always)"]
    G --> I{coefficient_expansion_strict or coefficient_expansion_lenient?}
    I -->|selected| I1[Coefficient Expansion - incl. normalization and zero-coefficient removal]
    I -->|skipped| J
    I1 --> J[End: Cleaned Results]

    style A fill:#e1f5ff
    style J fill:#e7f5e1
    style F fill:#fff4e1
    style G fill:#fff4e1

Process Stages¶

1. Abbreviation Filtering (optional — abbreviation_filtering)¶

Drops composition keys containing 2+ consecutive capital letters (abbreviations, junk keys).

2. Element Validation (optional — element_validation_strict)¶

Verifies compositions contain only valid periodic elements.

3. Element Validation (Lenient) (optional — element_validation_lenient)¶

Keeps a composition if it contains at least one embedded formula fragment anywhere in the text (letters-only runs, split at digits/punctuation, that individually parse as element symbols), instead of requiring the whole key to be pure elements. Drops only compositions with no recognizable formula fragment anywhere. Has no additional effect when element_validation_strict is also selected: the stricter result wins.

4. Text Normalization (optional — text_normalization)¶

Strips leading/trailing whitespace, collapses runs of multiple spaces down to one, and title-cases descriptive word tokens, leaving formula segments (tokens with digits, or tokens that fully parse as element symbols) and all-caps abbreviations untouched.

5. Miller Indices Filtering (optional — miller_indices)¶

Drops any composition entry carrying a crystal plane notation like (002), (111), (100), etc. entirely (not just the notation). Runs before Unicode/arithmetic resolution, see the following note.

6. Unicode Conversion (always runs)¶

Converts subscript Unicode characters to regular digits for standardization.

7. Arithmetic Resolution (always runs)¶

Evaluates mathematical expressions and fractional compositions.

8. Coefficient Expansion (optional — coefficient_expansion_strict / coefficient_expansion_lenient)¶

Expands coefficient patterns in chemical formulas, including:

• Leading coefficients: Multiplies all elements inside parentheses by leading coefficient
• Trailing coefficients: Multiplies all elements inside parentheses by trailing coefficient
• Parenthetical coefficients: Expands nested brackets with complex coefficient multiplication

Both coefficient_expansion_strict and coefficient_expansion_lenient trigger this same expansion logic. They differ only in what happens to compositions still carrying brackets/* afterward: coefficient_expansion_strict drops any of them as unresolved, while coefficient_expansion_lenient (when selected without coefficient_expansion_strict) spares compositions whose leftover brackets are balanced and contain genuine text rather than a failed arithmetic expression.

It also normalizes trailing zeros and removes zero-coefficient elements internally as part of expansion. There are no separate steps for those.

Examples of Various Cleaning Steps¶

Element Validation¶

Keeps only compositions that resolve to elements, or, in the lenient variant, contain at least one embedded formula fragment:

Selecting both element_validation_strict and element_validation_lenient together reverts to the strict column above.

Text Normalization Examples¶

Normalizes whitespace and title-cases descriptive words; formula segments are untouched:

Miller Indices Filtering¶

Drops any composition entry carrying a crystal plane notation entirely. It is not stripped down to the bare formula, since that would collapse distinct surface-orientation entries for the same material onto the same key:

Coefficient Expansion¶

Expands coefficients exactly like coefficient_expansion_strict, but keeps compositions whose leftover brackets are balanced and contain genuine text instead of dropping them as unresolved:

Selecting both coefficient_expansion_strict and coefficient_expansion_lenient together reverts to the strict column above.

Next Steps¶

• Learn about Evaluation
• Explore Visualization
• Configure Advanced RAG