Read, query, analyse and write MARCXML while preserving the structure of complete MARC records.
marcxmlr converts MARCXML to a predictable 11-column canonical table for
analysis with R, Arrow and other tabular tools. It can also select complete
records before materialisation using a native MARC-aware query engine, or
apply the same query afterwards to a tibble or Arrow Dataset. Large
catalogues can be streamed to Parquet, with file-level parallel processing for
collections distributed across multiple XML files. The canonical data can be
validated, edited and written back to MARCXML.
The specialised query interface is optional: it helps with MARC-aware record
selection without limiting subsequent analysis using dplyr, data.table,
SQL or other tools.
Explore the package:
MARC 21 stands for Machine-Readable Cataloging and is a family of communication formats used to represent and exchange bibliographic, authority, holdings, classification and community information. The examples in this README use bibliographic MARC 21.
MARCXML is the XML representation of MARC 21. It expresses the same logical record structure through XML elements and attributes rather than through the compact traditional MARC serialization.
marcxmlr converts MARCXML into a stable, documented, 11-column canonical
representation that preserves the represented MARC structure. Rather than
producing a task-specific extraction, it creates a reusable long-form table
that can be queried repeatedly as analytical questions evolve.
The package is implemented in R and read_marcxml() returns a tibble, but the
canonical data model is independent of R and dplyr. The examples use dplyr
for convenience: the same columns can be analysed with data.table and other
R tools, or stored in interoperable formats such as Parquet for use with Python,
Julia, Rust, SQL engines and other environments.
Install the current CRAN release with:
install.packages("marcxmlr")
The MARC-aware query interface (marc_has(), marc_filter_records() and
filter = during ingestion) requires version 0.5.0 or later. If CRAN
still offers an earlier version, install the version from GitHub:
# install.packages("pak")
# pak::pak("larry77/marcxmlr")
install.packages("marcxmlr", repos = c('https://larry77.r-universe.dev', 'https://cloud.r-project.org'))
Source installation from GitHub requires a C toolchain and libxml2 development
headers and libraries. On Debian and Ubuntu these are normally provided by
r-base-dev and libxml2-dev; Windows source builds use Rtools. Native
record prefiltering additionally requires Expat support at package build time
(for example, libexpat1-dev on Debian or Ubuntu). Other parsing and writing
features remain available on builds without Expat.
read_marcxml() converts an XML catalogue into a canonical tibble with 11
columns. This is an ordinary R table, but one MARC record occupies multiple
rows: its Leader, control fields and data-field subfields are represented
separately, tied together by record_id. The structure is explained in the
next section.
The bundled GPO sample contains 1,000 bibliographic records. Here we select
those containing the subject heading Flood insurance (650$a):
library(marcxmlr)
file <- system.file("extdata", "gpo-cgp-part00-sample.xml", package = "marcxmlr")
q <- marc_has("650", subfield = "a", equals = "Flood insurance")
# Filter the XML before converting the matching records to tabular rows.
selected <- read_marcxml(file, filter = q)
dim(selected) # 1,320 rows, 11 columns
#> [1] 1320 11
length(unique(selected$record_id))
#> [1] 28
# The very same query works on an already materialised canonical table.
canonical <- read_marcxml(file)
dim(canonical)
#> [1] 53808 11
selected_again <- marc_filter_records(canonical, q)
nrow(selected_again)
#> [1] 1320
Both paths select the same 28 complete records, not just the 650$a rows
that matched the query. These are alternatives: filter during ingestion to
avoid converting unwanted records, or filter after ingestion if you already
have the canonical table. In either case the result is ordinary tabular data,
ready for further analysis with dplyr or other tools. The full querying
interface, including Boolean combinations and Arrow Datasets, is explained
below.
A MARC record is not a conventional rectangular observation with one value for each variable. It is an ordered structured record. In its traditional machine-readable form, a record contains a Leader, a Directory, and an ordered sequence of variable fields. The Directory is structural bookkeeping for the compact MARC serialization: it records where the variable fields occur in the record.
For analytical purposes, the important content-bearing structure is easier to see by distinguishing three cases:
001 or 008, which contain a value but no
indicators or subfields; and100, 245, 264, 650 or 856, which contain a
three-digit tag, two indicators, and an ordered sequence of coded subfields.Both fields and subfields can repeat.
In conventional MARC notation, a title statement might look like:
245 10 $a Nineteen eighty-four / $c George Orwell.
Here 245 is the field tag, 1 and 0 are the two indicators, $a contains
the main title information, and $c contains the statement of responsibility.
The meaning of a particular tag, indicator or subfield comes from MARC 21; for
example, the Library of Congress documents these definitions in the
MARC 21 Format for Bibliographic Data.
The same field in MARCXML is:
<datafield tag="245" ind1="1" ind2="0">
<subfield code="a">Nineteen eighty-four /</subfield>
<subfield code="c">George Orwell.</subfield>
</datafield>
The XML hierarchy carries information that matters. The two <subfield>
elements are children of one particular <datafield> instance, and their order
is explicit. Repeated sibling <datafield> elements remain distinct even if
they have the same tag and indicators.
A subject field gives another example:
<datafield tag="650" ind1=" " ind2="0">
<subfield code="a">Totalitarianism</subfield>
<subfield code="v">Fiction.</subfield>
</datafield>
The blank first indicator is an actual MARC value. It is not missing data. A representation that intends to preserve MARC faithfully has to distinguish that blank from a case where indicators do not apply at all, as for a Leader or control field.
A tempting analytical representation of a bibliographic record is one row per record:
| record_id | title | author | subject |
|---|---|---|---|
| 1 | Nineteen eighty four | George Orwell | Totalitarianism |
Such a table may be exactly what one particular analysis needs, but it is not a general representation of the MARC record.
Consider two distinct subject fields:
650 _0 $a Totalitarianism $v Fiction
650 _0 $a Dystopias $v Fiction
These are two different 650 field instances. Each $v Fiction belongs to one
particular occurrence of 650.
If values are flattened independently by tag and subfield code, we might obtain:
650$a = "Totalitarianism || Dystopias"
650$v = "Fiction || Fiction"
All values are still present, but field membership has disappeared. With more heterogeneous repeated fields, the original associations cannot in general be reconstructed unambiguously.
The same problem appears inside a single field when a subfield code repeats:
856 40 $u https://example.org/1984 $y Full text $y Mirror
The two $y values are two ordered occurrences inside one specific 856 field.
A faithful general representation therefore has to preserve several levels of
identity and order:
record
↓
field instance
↓
field tag, indicators and field position
↓
ordered subfield instances
↓
subfield code, value, order and occurrence
This is the representational problem addressed by marcxmlr. The package does
not force MARC into a task-specific wide table during import. Instead, it first
creates a rectangular representation in which the original relationships remain
explicit. The analyst can then simplify that representation deliberately when
the purpose of the analysis is known.
Turning MARCXML into a table is not simply a matter of extracting values. A
useful representation must preserve the relationships that give MARC 21 records
their meaning while remaining accessible to people and software that work with
ordinary tabular data. Four requirements guide the design of marcxmlr:
dplyr, data.table,
other R tools, or software in Python, Julia, Rust and SQL environments via
interoperable tabular formats, including Parquet (subject to each tool's
capabilities and scale limits).245$a or
856$u with ordinary table filters, or use marc_has() to select complete
records before or after materialisation. See the MARC-aware query
interface.These are design requirements, not a claim of mathematical optimality. A more compact representation might store complete fields in nested cells, distribute the hierarchy across related tables or encode field boundaries through row sequences. Such designs may be useful for particular purposes, but they make ordinary MARC-oriented queries or exchange with generic tabular tools less direct. Compactness alone is therefore not a sufficient measure of quality.
The canonical representation is marcxmlr's practical answer to these
requirements: a long, fixed-schema table with 11 scalar-valued columns. Each
Leader and control field occupies one row; each data-field subfield occupies
one row with its field context and structural coordinates. This makes familiar
MARC-oriented filtering, grouping and inspection possible with ordinary tabular
operations. The 11-column specification and the examples
below make this mapping explicit.
The design accepts controlled logical redundancy: field tags, indicators and other field-level information repeat across rows belonging to the same data field. This is deliberate. It allows each subfield to remain intelligible without joins or special decoding, while retaining the relationships needed for reconstruction.
Logical redundancy does not imply equally inefficient physical storage. Parquet supports dictionary and run-length encodings and compression, which can substantially reduce the storage cost of repeated column values. A compressed CSV file (for example, CSV with gzip compression) can also exploit repeated patterns. Actual file sizes depend on the data and the compression settings; no claim is made that this representation is always the smallest. See the Parquet workflow for out-of-memory storage.
The canonical model is neither the only possible lossless tabular mapping nor claimed to be universally optimal. It is a deliberate compromise: alternatives that eliminate redundancy tend to introduce other costs in interpretability, interoperability or MARC-native querying. Improving substantially on this combination of properties is not straightforward.
The row meaning depends on the MARC structure being represented:
For a data field with several subfields, field-level information such as the tag, indicators and field position is repeated on every row belonging to that field. That repetition is deliberate. It lets each row carry enough context for direct analysis while still making it possible to identify which rows belong to the same field instance.
MARC tags remain values in the tag column rather than becoming columns of
their own. The schema therefore does not change when a catalogue contains an
unusual but structurally valid field. MARC knowledge determines what a field
means; the canonical representation determines how its structure is retained.
Every canonical result contains these columns, in this order:
| Column | Type | Meaning |
|---|---|---|
record_id |
integer | Sequential identity assigned by marcxmlr; distinct from MARC control field 001. |
field_type |
character |
leader, controlfield, or datafield. |
tag |
character |
LDR for the Leader, otherwise the MARC field tag. |
subfield_code |
character | MARC subfield code; NA for Leaders and control fields. |
value |
character | Textual value, preserving meaningful source content. |
field_order |
integer | Position of the field instance in the source record; the Leader is 0. |
field_occurrence |
integer | Occurrence number of the same field type and tag within the record. |
ind1 |
character | First data-field indicator; may be " " when blank; otherwise NA when not applicable. |
ind2 |
character | Second data-field indicator; may be " " when blank; otherwise NA when not applicable. |
subfield_order |
integer | Position of the subfield inside its containing data field; otherwise NA. |
subfield_occurrence |
integer | Occurrence number of that subfield code inside that particular field instance; otherwise NA. |
The columns are easier to understand as four related groups.
Identity and structural kind. record_id groups every canonical row produced
from the same MARC record. It is deliberately distinct from control field 001,
which is source metadata and may be absent, nonnumeric or duplicated across
files. field_type distinguishes Leaders, control fields and data fields.
MARC content. tag, subfield_code, value, ind1 and ind2 carry the
content designation and values present in the source. For Leaders and control
fields, subfield and indicator columns are NA because those concepts do not
apply. A blank data-field indicator is instead preserved as the literal
one-character string " ".
Structural coordinates. record_id, field_order and subfield_order are
the principal coordinates of the hierarchy. Rows from the same data field share
the same record_id and field_order; subfield_order preserves the sequence
inside that field. These coordinates retain field membership and source order
and are the coordinates needed for faithful reconstruction.
Analytical occurrence coordinates. field_occurrence and
subfield_occurrence are derived conveniences for analysis. They number
repetition directly, making questions such as “the second 650 in this record”
or “the second $y in this 856” easy to state. They are not the authoritative
source of serialization order; field_order and subfield_order already encode
that structure.
The small amount of redundancy is intentional. Repeating field context and storing occurrence numbers makes the table easier to inspect, filter, group and join. The package favours an explicit analyst-friendly representation over a minimally encoded one.
The package bundles the following synthetic record as
inst/extdata/orwell-marcxml.xml. Every example below refers to this same record.
<record xmlns="http://www.loc.gov/MARC21/slim">
<leader>00000cam a2200000 i 4500</leader>
<controlfield tag="001">12345</controlfield>
<datafield tag="100" ind1="1" ind2=" ">
<subfield code="a">Orwell, George,</subfield>
<subfield code="d">1903-1950.</subfield>
</datafield>
<datafield tag="245" ind1="1" ind2="0">
<subfield code="a">Nineteen eighty-four /</subfield>
<subfield code="c">George Orwell.</subfield>
</datafield>
<datafield tag="264" ind1=" " ind2="1">
<subfield code="a">London :</subfield>
<subfield code="b">Secker & Warburg,</subfield>
<subfield code="c">1949.</subfield>
</datafield>
<datafield tag="650" ind1=" " ind2="0">
<subfield code="a">Totalitarianism</subfield>
<subfield code="v">Fiction.</subfield>
</datafield>
<datafield tag="650" ind1=" " ind2="0">
<subfield code="a">Dystopias.</subfield>
<subfield code="v">Fiction.</subfield>
</datafield>
<datafield tag="856" ind1="4" ind2="0">
<subfield code="u">https://example.org/1984</subfield>
<subfield code="y">Full text</subfield>
<subfield code="y">Mirror</subfield>
</datafield>
</record>
In canonical form, that one record becomes 16 rows and exactly 11 columns. For
readability, <blank> below denotes the actual one-character indicator value
" ":
| record_id | field_type | tag | subfield_code | value | field_order | field_occurrence | ind1 | ind2 | subfield_order | subfield_occurrence |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | leader | LDR | NA | 00000cam a2200000 i 4500 | 0 | 1 | NA | NA | NA | NA |
| 1 | controlfield | 001 | NA | 12345 | 1 | 1 | NA | NA | NA | NA |
| 1 | datafield | 100 | a | Orwell, George, | 2 | 1 | 1 | <blank> |
1 | 1 |
| 1 | datafield | 100 | d | 1903-1950. | 2 | 1 | 1 | <blank> |
2 | 1 |
| 1 | datafield | 245 | a | Nineteen eighty-four / | 3 | 1 | 1 | 0 | 1 | 1 |
| 1 | datafield | 245 | c | George Orwell. | 3 | 1 | 1 | 0 | 2 | 1 |
| 1 | datafield | 264 | a | London : | 4 | 1 | <blank> |
1 | 1 | 1 |
| 1 | datafield | 264 | b | Secker & Warburg, | 4 | 1 | <blank> |
1 | 2 | 1 |
| 1 | datafield | 264 | c | 1949. | 4 | 1 | <blank> |
1 | 3 | 1 |
| 1 | datafield | 650 | a | Totalitarianism | 5 | 1 | <blank> |
0 | 1 | 1 |
| 1 | datafield | 650 | v | Fiction. | 5 | 1 | <blank> |
0 | 2 | 1 |
| 1 | datafield | 650 | a | Dystopias. | 6 | 2 | <blank> |
0 | 1 | 1 |
| 1 | datafield | 650 | v | Fiction. | 6 | 2 | <blank> |
0 | 2 | 1 |
| 1 | datafield | 856 | u | https://example.org/1984 | 7 | 1 | 4 | 0 | 1 | 1 |
| 1 | datafield | 856 | y | Full text | 7 | 1 | 4 | 0 | 2 | 1 |
| 1 | datafield | 856 | y | Mirror | 7 | 1 | 4 | 0 | 3 | 2 |
Several features are visible directly:
650 fields have the same tag but different field_order and
field_occurrence values;650 share the same field_order, so $a and $v
remain attached to the correct field instance;856 $y values remain distinct, with subfield_occurrence values 1
and 2;For someone who knows MARC 21, catalogue questions translate naturally into conditions on documented columns. In plain language:
| Catalogue question | MARC designation | Condition on the canonical table |
|---|---|---|
| What is the main title? | 245 $a |
tag is 245 and subfield code is a
|
| Which publisher is named in the publication statement? | 264 #1 $b |
tag is 264, second indicator is 1, and subfield code is b
|
| Which is the second subject field? | second 650
|
tag is 650 and field occurrence is 2
|
| Which is the second link label? | second 856 $y
|
tag is 856, subfield code is y, and subfield occurrence is 2
|
The package does not need to know that 245 is a title field or that
264 #1 $b identifies a publisher. That semantic knowledge comes from MARC 21.
marcxmlr provides a stable representation in which the corresponding
conditions can be expressed with ordinary tabular operations.
This is why the representation is deliberately richer than many final analytical products. It preserves relationships before the analyst decides which ones can safely be discarded. Canonical to simplified is easy; simplified to canonical may be impossible.
marcxmlr preserves the represented MARC structure and values, not the
incidental byte-level serialization of the XML file. A read/write cycle can
preserve Leaders, control fields, data-field instances, indicators, subfields,
repetition, values and ordering without reproducing the same indentation,
namespace-prefix spelling, attribute order or other formatting details.
This preserve-first principle is the package's main design choice: field membership, order or repetition discarded during import may be impossible to reconstruct unambiguously afterward.
All examples below use the exact record shown above. It is bundled with the package, so nothing needs to be downloaded.
library(marcxmlr)
library(dplyr)
source_xml <- system.file(
"extdata",
"orwell-marcxml.xml",
package = "marcxmlr"
)
marc <- read_marcxml(source_xml)
dim(marc)
#> [1] 16 11
The result is an ordinary tibble with the canonical columns described above.
Now the plain-language conditions can be translated directly into dplyr.
MARC 21 says that the main title is in 245 $a: tag is 245 and subfield code
is a.
marc |>
filter(tag == "245", subfield_code == "a") |>
select(record_id, value)
#> # A tibble: 1 × 2
#> record_id value
#> <int> <chr>
#> 1 1 Nineteen eighty-four /
For this record, 264 #1 $b identifies the publisher: tag is 264, the second
indicator is 1, and the subfield code is b.
marc |>
filter(tag == "264", ind2 == "1", subfield_code == "b") |>
select(record_id, value)
#> # A tibble: 1 × 2
#> record_id value
#> <int> <chr>
#> 1 1 Secker & Warburg,
The important transition is from a catalogue question, to its MARC content designation, to ordinary table columns. XML traversal is no longer part of the analysis.
650 $a Totalitarianism?Sometimes the row that matches is not the whole answer. Here the question is
not only to find $a Totalitarianism, but to keep the complete 650 field to
which it belongs. Rows from one data-field instance share record_id and
field_order, so the field can be selected as a group:
marc |>
group_by(record_id, field_order) |>
filter(
any(
tag == "650" &
subfield_code == "a" &
value == "Totalitarianism"
)
) |>
ungroup() |>
select(record_id, field_order, tag, subfield_code, value)
#> # A tibble: 2 × 5
#> record_id field_order tag subfield_code value
#> <int> <int> <chr> <chr> <chr>
#> 1 1 5 650 a Totalitarianism
#> 2 1 5 650 v Fiction.
The condition matches only $a Totalitarianism, but the result also contains
its associated $v Fiction. because both rows belong to the same field.
The two subject fields remain separate:
marc |>
filter(tag == "650", subfield_code == "a") |>
select(record_id, field_order, field_occurrence, value)
#> # A tibble: 2 × 4
#> record_id field_order field_occurrence value
#> <int> <int> <int> <chr>
#> 1 1 5 1 Totalitarianism
#> 2 1 6 2 Dystopias.
Repetition inside one field is equally explicit:
marc |>
filter(tag == "856") |>
select(subfield_code, value, subfield_order, subfield_occurrence)
#> # A tibble: 3 × 4
#> subfield_code value subfield_order subfield_occurrence
#> <chr> <chr> <int> <int>
#> 1 u https://example.org/1984 1 1
#> 2 y Full text 2 1
#> 3 y Mirror 3 2
The canonical representation preserves structure first. A simpler table can
then be derived deliberately for a particular analysis. For example, keep one
title and collapse each 650 field to a readable subject string:
titles <- marc |>
filter(tag == "245", subfield_code == "a") |>
transmute(record_id, title = value)
subjects <- marc |>
filter(tag == "650") |>
group_by(record_id, field_order) |>
summarise(
subject = paste(value, collapse = " ; "),
.groups = "drop"
) |>
group_by(record_id) |>
summarise(
subjects = paste(subject, collapse = " | "),
.groups = "drop"
)
left_join(titles, subjects, by = "record_id")
#> # A tibble: 1 × 3
#> record_id title subjects
#> <int> <chr> <chr>
#> 1 1 Nineteen eighty-four / Totalitarianism ; Fiction. | Dystopias. ; Fiction.
This is the intended asymmetry: once the MARC structure is preserved, it is easy to collapse it for a particular purpose. Reconstructing field membership after it has already been discarded may be impossible.
Because subfield_occurrence makes repetition explicit, a particular repeated
value can be changed without affecting its neighbour. Here only the second
856 $y changes:
edited <- marc |>
mutate(
value = if_else(
record_id == 1L &
tag == "856" &
subfield_code == "y" &
subfield_occurrence == 2L,
"Backup access",
value
)
)
edited |>
filter(tag == "856") |>
select(subfield_code, value, subfield_occurrence)
#> # A tibble: 3 × 3
#> subfield_code value subfield_occurrence
#> <chr> <chr> <int>
#> 1 u https://example.org/1984 1
#> 2 y Full text 1
#> 3 y Backup access 2
After filtering or editing canonical data, diagnose_canonical() can inspect
whether the representation still satisfies the structural contract required for
safe serialization.
diagnostics <- diagnose_canonical(edited)
diagnostics
#> # A tibble: 0 × 6
#> # ℹ 6 variables: severity <chr>, code <chr>, message <chr>, record_id <int>,
#> # field_order <int>, subfield_order <int>
An empty diagnostics tibble means that no structural issue was found. The
function checks the canonical representation used by marcxmlr; it is not a
complete MARC 21 content validator and does not replace MARCXML schema
validation.
For Arrow datasets and lazy Arrow queries, diagnose_canonical()
validates the representation incrementally without collecting the entire
dataset into memory. Records must appear in non-decreasing record_id
order. Gaps in record identifiers may produce warnings without indicating
structural errors.
write_marcxml() performs the required structural checks itself before writing.
Calling diagnose_canonical() explicitly is useful when data have been filtered
or edited because the result can be inspected before any XML file is created.
A structurally coherent canonical table can be serialized again:
out_xml <- tempfile(fileext = ".xml")
write_marcxml(edited, out_xml)
reimported <- read_marcxml(out_xml)
identical(edited, reimported)
#> [1] TRUE
Writing the table to MARCXML and reading it again preserves the MARC semantics represented by the canonical table, not incidental XML formatting such as indentation or attribute order.
A complete record can likewise be selected and exported with ordinary table operations before writing.
By default, read_marcxml() stops when it encounters a structurally malformed
MARC record. Alternatively, on_marc_error = "skip" skips the entire malformed
record and continues processing. An error report is required in this mode:
marc <- read_marcxml(
source_xml,
on_marc_error = "skip",
error_report = "marc-errors.csv"
)
The same options are available in marcxml_to_parquet(). The report identifies
skipped records and includes the diagnostic reason and original record XML.
Source-position gaps in record_id are preserved.
Recovery applies only to MARC-level structural errors. XML syntax and encoding errors remain fatal; the package does not silently repair malformed XML.
Both the in-memory and Parquet workflows use the same 11-column canonical representation. The practical difference is where that representation lives.
Use read_marcxml() when the source and expanded canonical result fit
comfortably in memory. The n_max argument can restrict parsing to the first
records, which is useful for previews and development:
preview <- read_marcxml(
source_xml,
n_max = 1L
)
unique(preview$record_id)
#> [1] 1
For larger collections, materializing all canonical rows as one R object may be
unnecessary. marcxml_to_parquet() writes the same schema to a directory of
Parquet files while keeping the expanded canonical data bounded by processing
batches.
Arrow support is optional. Install it when needed:
install.packages("arrow")
The bundled example can be converted to Parquet with:
parquet_dir <- tempfile()
marcxml_to_parquet(
source_xml,
output_dir = parquet_dir,
workers = 1L,
verbose = FALSE
)
catalogue <- arrow::open_dataset(parquet_dir)
catalogue |>
summarise(rows = n()) |>
collect()
#> # A tibble: 1 × 1
#> rows
#> <int>
#> 1 16
The dataset can be filtered and aggregated with Arrow and dplyr before a
smaller result is collected into R:
catalogue |>
filter(tag == "650", subfield_code == "a") |>
count(value, sort = TRUE) |>
collect()
For a catalogue already split across several files:
files <- sort(Sys.glob("data/catalogue/*.xml"))
marcxml_to_parquet(
files,
output_dir = "catalogue_parquet",
workers = 4L
)
When several input files are converted together, record_id values are assigned
in deterministic input-file order and remain globally contiguous across the
logical dataset.
For one MARCXML file, the optimized sequential path is the natural starting point and additional worker processes may be slower. Parallel execution is more naturally useful when substantial independent files can be assigned to separate workers.
MARC records contain many rows in the canonical representation. A conventional
dplyr::filter() selects rows; sometimes the desired result is instead
every field of each record satisfying a bibliographic condition. The two
new public functions address that distinction:
marc_has() defines a MARC-aware, record-level query; it does not
immediately evaluate it.marc_filter_records() applies that query to an existing canonical
tibble or Arrow Dataset, retaining all rows of selected records.During XML ingestion, the same query is supplied through the filter argument
of read_marcxml() or marcxml_to_parquet(). Native Expat screening identifies
matching records, and the native libxml2 reader materialises only those complete
records. For restrictive queries this avoids materialising most of the
catalogue before deciding what to keep.
marc_has() supports a field tag (including "LDR" for the Leader), an
optional subfield, equals for a case-sensitive exact value, contains for
a case-sensitive literal substring, and optional ind1 and ind2.
equals and contains cannot be combined in the same call.
# Any record with an electronic location field.
marc_has("856")
# An electronic URL containing a particular domain.
marc_has("856", subfield = "u", contains = "europa.eu")
# A subject field with specified indicators and a controlled-vocabulary code.
marc_has("650", subfield = "2", equals = "EUROVOC",
ind1 = "0", ind2 = "7")
Queries can be combined with ordinary R operators & (AND), | (OR), and
! (NOT), including nested expressions. The operators work at record
level, so different marc_has() conditions may match different fields or
subfields of the same record:
q <- (marc_has("650", subfield = "a", contains = "Economics") |
marc_has("650", subfield = "a", contains = "Trade")) &
marc_has("856", subfield = "u") &
!marc_has("999")
Within one marc_has() call, the specified conditions must match the same
canonical row. By contrast, two calls combined with & need only match the
same record: they are not required to refer to the same occurrence of a
repeated data field. Occurrence-specific relationships can be examined using
field_order and subfield_order after materialisation.
Unlike ordinary row filtering, a conjunction can match conditions in
different fields of the same record. In the GPO sample, we can select
records about flood insurance that also contain a 710 corporate added-entry
field:
gpo_xml <- system.file(
"extdata", "gpo-cgp-part00-sample.xml", package = "marcxmlr"
)
q <- marc_has("650", subfield = "a", equals = "Flood insurance") &
marc_has("710")
matches <- read_marcxml(gpo_xml, filter = q)
c(records = length(unique(matches$record_id)), rows = nrow(matches))
#> records rows
#> 9 466
The 650$a and 710 conditions need not occur in the same canonical row;
they only need to belong to the same MARC record. All fields of the nine
matching records are retained. Queries can also use |, ! and parentheses,
as illustrated above.
One identity detail matters when comparing the two execution paths:
filtering during ingestion renumbers selected records consecutively,
whereas filtering an existing tibble preserves its original record_id values.
Selection semantics are the same, but those numeric IDs need not match.
The same query can be used to write only selected records to Parquet (requires
arrow):
out <- tempfile("gpo-selected-parquet-")
marcxml_to_parquet(
gpo_xml,
output_dir = out,
filter = q,
batch_records = 200L,
workers = 1L,
verbose = FALSE
)
# Query the materialised Parquet Dataset again without collecting it first.
ds <- arrow::open_dataset(out)
records_with_subjects <- ds |>
marc_filter_records(marc_has("650", subfield = "a")) |>
dplyr::collect()
The Arrow-backed result stays lazy until collect(); supported filters and
joins are translated to Arrow operations. record_id must identify records
uniquely across the entire Dataset, including all Parquet fragments.
For multiple input XML files, the same native query can be evaluated independently by file-level workers:
marcxml_to_parquet(
c(source_xml, gpo_xml),
output_dir = tempfile("selected-catalogue-"),
filter = marc_has("856"),
workers = 2L,
verbose = FALSE
)
Selected records receive globally unique, consecutive record_id values in
input-file order within that conversion. Parallel filtering within one XML
file is deliberately unsupported; the supported filtered parallel workflow
assigns different input files to different workers.
marc_has() is a convenience for selecting complete records, not a general
purpose expression evaluator. Arbitrary dplyr predicates cannot run in the
native XML prefilter, and marc_filter_records() accepts MARC query objects
rather than raw dplyr expressions. After materialisation, however, the
canonical table remains an ordinary tibble: users can apply dplyr, regular
expressions, custom joins, aggregations or their own analysis without using
marc_has() at all. Arrow-backed datasets likewise support the operations
that Arrow can translate.
The original filter_tag, filter_subfield and filter_equals arguments are
still available for backward compatibility:
read_marcxml(
gpo_xml,
filter_tag = "856"
)
Do not combine these legacy arguments with filter = q. Both native filtering
interfaces require uncompressed UTF-8 MARCXML without a DTD and an
installation built with Expat support. Filtering cannot be combined with
MARC-error recovery or chunk_records. Calls without filtering retain the
original unfiltered execution paths. If no record matches, the result has the
canonical schema with zero rows (or an empty canonical Parquet Dataset).
The in-memory and Parquet workflows use the same logical representation and, for supported ordinary unfiltered MARCXML, the same package-specific compiled parser. The important difference is how much of the expanded canonical representation is materialized at one time. For filtered XML, Expat first identifies the matching records and their byte offsets; the native reader then materializes those complete records using the same canonical schema.
The optimized sequential unfiltered parser is implemented in C on top of
libxml2 and uses two passes over the input. The first pass determines how many canonical rows
each selected record will produce. The second pass allocates the required output
vectors at their final size and fills them directly while traversing the MARC
structure in source order.
For read_marcxml(), the second pass materializes the complete selected result
as one tibble. For marcxml_to_parquet(), successive groups of complete records
are materialized and written as Parquet parts, so the complete expanded table
does not have to exist as one R object. The record remains the natural processing
unit because its Leader, fields and subfields must remain structurally coherent.
The same principle applies in the other direction. write_marcxml() can write
an in-memory canonical table directly, while Arrow-backed canonical data can be
scanned incrementally without first collecting the complete Dataset into one
R tibble. Structural relationships are reconstructed from record_id,
field_order and subfield_order; the occurrence columns remain analytical
coordinates rather than the authoritative source of serialization order.
Readers who want the lower-level details of the libxml2 reader/writer,
two-pass allocation, bounded Arrow scans and retained reference implementations
can continue with Technical implementation and performance.
workers = 1L is the default and should normally be the starting point for a
single MARCXML file. With one file, requesting multiple workers requires a
worker-safe record-based execution path and adds process creation, task
coordination and data-transfer overhead. Once the sequential parser itself is
fast, those costs can outweigh the available parallel work.
A catalogue already divided into several substantial MARCXML files is a more
natural parallel workload. Complete files can be assigned to separate workers,
while each worker retains the optimized sequential parser internally. Before
multi-file conversion begins, marcxmlr assigns deterministic, non-overlapping
record_id ranges from input-file order, so worker completion order does not
change the logical record identities in the resulting Dataset.
This is a practical recommendation rather than a universal threshold: CPU, memory, storage, compression, file layout and record complexity all matter. Benchmark representative input before relying on additional workers.
The following measurements illustrate the scale of unfiltered reading,
Parquet conversion and MARCXML writing. The underlying execution paths remain
part of the current package, including version 0.5.0. The measurements were
originally collected with marcxmlr 0.3.1 on a Lenovo ThinkPad P50 with an
Intel Core i7-6820HQ, 31.18 GiB RAM and a local SSD, using R 4.6.1 and
Arrow 25.0.1; they have not been rerun as a 0.5.0 benchmark campaign.
Timings indicate scale, not guaranteed performance on other systems.
| Workflow | Input | Median elapsed time |
|---|---|---|
read_marcxml() |
40,000 records / 2,143,952 rows | 7.106 s |
marcxml_to_parquet(), 1 worker |
40,000 records / 2,143,952 rows | 9.374 s |
write_marcxml() from Arrow |
40,000 records / 2,143,952 rows | 14.493 s |
marcxml_to_parquet(), 4 workers |
27 files / 1,080,000 records / 67,672,396 rows | 118.629 s |
The single-file parallel measurements are also a useful warning against assuming that more workers are automatically faster: the same 40,000-record Parquet conversion took 19.858 s with two workers and 18.849 s with four, compared with 9.374 s sequentially. Across the 27-file corpus, by contrast, four workers reduced elapsed time from 290.139 s to 118.629 s.
Detailed benchmark methodology, memory measurements and implementation context are provided in Technical implementation and performance.
marcxmlr deliberately separates faithful structural representation from later
interpretation. It does not attempt to:
Its job is to make supported MARCXML available through a stable, structurally faithful canonical table, with optional efficient selection of complete records before or after materialisation. Subsequent reshaping, mapping and analysis remain choices made with general-purpose tools in R or elsewhere.
For further reading:
The README and vignette are intended to be sufficient for using the package. The technical note is supplementary material for readers who want to understand how the computational paths are implemented and evaluated.
The implementation principle is the same throughout: preserve the MARC structure first, then simplify or optimize without changing the logical data model.