2025-12-06 01:19:41,022 ocrqa_create_bloom_filter.py:282 INFO: Git version: 293abfb
2025-12-06 01:19:41,022 ocrqa_create_bloom_filter.py:284 INFO: Using normalization version: 2
2025-12-06 01:19:41,022 ocrqa_create_bloom_filter.py:286 INFO: Language-specific handling: lb
2025-12-06 01:19:41,022 ocrqa_create_bloom_filter.py:288 INFO: Frequency threshold mode: adaptive
2025-12-06 01:19:41,022 ocrqa_create_bloom_filter.py:290 INFO: Adaptive parameters: base_freq=500.0, exponent=2.0, min=2
2025-12-06 01:19:41,022 ocrqa_create_bloom_filter.py:1063 INFO: Namespace(input_files=['lex/lb/lbwiki.unigram.freq.tsv.bz2', 'lex/lb/wortschatz.freq.tsv.bz2', 'lex/lb/modern_spelling.rw.txt', 'lex/lb/old_spelling.rw.txt', 'lex/lb/ocr_errors.nw.txt', 'lex/lb/realword_ocr_errors.nw.txt'], bloom_path='build.d/fp_prob_0.00005_base_500.0/ocrqa-wp_v2.0.0-lb.bloom', fp_probability=5e-05, log_level='INFO', log_file='build.d/fp_prob_0.00005_base_500.0/ocrqa-wp_v2.0.0-lb.bloom.log', config=None, min_frequency=2, single_char_min_frequency=500, frequency_threshold_mode='adaptive', adaptive_base_freq=500.0, adaptive_exponent=2.0, diagnose_bloom=True, git_version='293abfb', log_lexicon='build.d/fp_prob_0.00005_base_500.0/ocrqa-wp_v2.0.0-lb.bloom.lexicon.txt', version=2, language='lb')
2025-12-06 01:19:41,023 ocrqa_create_bloom_filter.py:606 INFO: Starting Bloom Filter creation...
2025-12-06 01:19:41,023 ocrqa_create_bloom_filter.py:469 INFO: Processing nonword file: lex/lb/ocr_errors.nw.txt
2025-12-06 01:19:41,023 ocrqa_create_bloom_filter.py:495 INFO: Exclusion list built: 20 words will be filtered from bloom filter
2025-12-06 01:19:41,023 ocrqa_create_bloom_filter.py:469 INFO: Processing nonword file: lex/lb/realword_ocr_errors.nw.txt
2025-12-06 01:19:41,023 ocrqa_create_bloom_filter.py:495 INFO: Exclusion list built: 22 words will be filtered from bloom filter
2025-12-06 01:19:41,023 ocrqa_create_bloom_filter.py:385 INFO: Processing frequency file: lex/lb/lbwiki.unigram.freq.tsv.bz2
2025-12-06 01:19:41,045 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 1: 500
2025-12-06 01:19:41,046 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 2: 125
2025-12-06 01:19:41,046 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 3: 55
2025-12-06 01:19:41,046 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 4: 31
2025-12-06 01:19:41,046 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 5: 20
2025-12-06 01:19:41,047 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 10: 5
2025-12-06 01:19:41,047 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 6: 13
2025-12-06 01:19:41,047 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 7: 10
2025-12-06 01:19:41,047 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 9: 6
2025-12-06 01:19:41,047 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 12: 3
2025-12-06 01:19:41,048 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 11: 4
2025-12-06 01:19:41,048 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 8: 7
2025-12-06 01:19:41,049 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 15: 2
2025-12-06 01:19:41,050 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 13: 2
2025-12-06 01:19:41,051 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 14: 2
2025-12-06 01:19:41,055 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 16: 2
2025-12-06 01:19:41,060 ocrqa_create_bloom_filter.py:421 WARNING: Skipping invalid line in frequency file lex/lb/lbwiki.unigram.freq.tsv.bz2: '446' (expected format: <count>\t<word>)
2025-12-06 01:19:41,063 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 17: 2
2025-12-06 01:19:41,064 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 22: 2
2025-12-06 01:19:41,073 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 18: 2
2025-12-06 01:19:41,074 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 19: 2
2025-12-06 01:19:41,075 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 23: 2
2025-12-06 01:19:41,082 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 21: 2
2025-12-06 01:19:41,086 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 20: 2
2025-12-06 01:19:41,087 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 29: 2
2025-12-06 01:19:41,103 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 24: 2
2025-12-06 01:19:41,112 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 28: 2
2025-12-06 01:19:41,135 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 25: 2
2025-12-06 01:19:41,141 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 31: 2
2025-12-06 01:19:41,148 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 26: 2
2025-12-06 01:19:41,208 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 30: 2
2025-12-06 01:19:41,224 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 27: 2
2025-12-06 01:19:41,249 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 40: 2
2025-12-06 01:19:41,250 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 33: 2
2025-12-06 01:19:41,269 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 36: 2
2025-12-06 01:19:41,277 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 32: 2
2025-12-06 01:19:41,366 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 58: 2
2025-12-06 01:19:41,388 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 37: 2
2025-12-06 01:19:41,460 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 34: 2
2025-12-06 01:19:41,480 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 35: 2
2025-12-06 01:19:41,583 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 38: 2
2025-12-06 01:19:41,584 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 42: 2
2025-12-06 01:19:41,655 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 66: 2
2025-12-06 01:19:41,886 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 39: 2
2025-12-06 01:19:41,931 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 51: 2
2025-12-06 01:19:41,946 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 43: 2
2025-12-06 01:19:41,997 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 49: 2
2025-12-06 01:19:42,058 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 41: 2
2025-12-06 01:19:42,101 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 62: 2
2025-12-06 01:19:42,288 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 65: 2
2025-12-06 01:19:42,339 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 44: 2
2025-12-06 01:19:42,431 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 45: 2
2025-12-06 01:19:42,478 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 52: 2
2025-12-06 01:19:42,496 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 46: 2
2025-12-06 01:19:42,518 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 54: 2
2025-12-06 01:19:42,552 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 50: 2
2025-12-06 01:19:42,568 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 61: 2
2025-12-06 01:19:42,728 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 47: 2
2025-12-06 01:19:42,757 ocrqa_create_bloom_filter.py:385 INFO: Processing frequency file: lex/lb/wortschatz.freq.tsv.bz2
2025-12-06 01:19:43,211 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 60: 2
2025-12-06 01:19:43,324 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 53: 2
2025-12-06 01:19:43,364 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 55: 2
2025-12-06 01:19:43,669 ocrqa_create_bloom_filter.py:421 WARNING: Skipping invalid line in frequency file lex/lb/wortschatz.freq.tsv.bz2: '' (expected format: <count>\t<word>)
2025-12-06 01:19:43,669 ocrqa_create_bloom_filter.py:528 INFO: Processing real-word file: lex/lb/modern_spelling.rw.txt
2025-12-06 01:19:44,039 ocrqa_create_bloom_filter.py:528 INFO: Processing real-word file: lex/lb/old_spelling.rw.txt
2025-12-06 01:19:44,040 ocrqa_create_bloom_filter.py:623 INFO: Low-frequency exclusions before cleanup: 230319 tokens
2025-12-06 01:19:44,057 ocrqa_create_bloom_filter.py:631 INFO: Low-frequency exclusions after cleanup: 176983 tokens
2025-12-06 01:19:44,057 ocrqa_create_bloom_filter.py:635 INFO: Lexical processing complete.
2025-12-06 01:19:44,057 ocrqa_create_bloom_filter.py:638 INFO: Processing Statistics:
2025-12-06 01:19:44,057 ocrqa_create_bloom_filter.py:640 INFO:   freq_words_filtered: 275,227
2025-12-06 01:19:44,057 ocrqa_create_bloom_filter.py:640 INFO:   freq_words_read: 487,375
2025-12-06 01:19:44,057 ocrqa_create_bloom_filter.py:640 INFO:   invalid_lines_filtered: 2
2025-12-06 01:19:44,057 ocrqa_create_bloom_filter.py:640 INFO:   low_freq_excluded: 176,983
2025-12-06 01:19:44,057 ocrqa_create_bloom_filter.py:640 INFO:   nonwords_count: 22
2025-12-06 01:19:44,057 ocrqa_create_bloom_filter.py:640 INFO:   nonwords_read: 22
2025-12-06 01:19:44,057 ocrqa_create_bloom_filter.py:640 INFO:   realwords_accepted: 152,113
2025-12-06 01:19:44,057 ocrqa_create_bloom_filter.py:640 INFO:   realwords_read: 152,113
2025-12-06 01:19:44,057 ocrqa_create_bloom_filter.py:640 INFO:   single_char_words_filtered: 469
2025-12-06 01:19:44,057 ocrqa_create_bloom_filter.py:640 INFO:   subwords_accepted: 257,197
2025-12-06 01:19:44,057 ocrqa_create_bloom_filter.py:640 INFO:   subwords_filtered: 17
2025-12-06 01:19:44,057 ocrqa_create_bloom_filter.py:640 INFO:   words_accepted: 211,677
2025-12-06 01:19:44,057 ocrqa_create_bloom_filter.py:644 INFO: Creating bloom filter with 251,359 unique words/tokens
2025-12-06 01:19:44,057 ocrqa_create_bloom_filter.py:647 INFO: Target false positive probability: 5e-05
2025-12-06 01:19:44,128 ocrqa_create_bloom_filter.py:652 INFO: Bloom filter created successfully: build.d/fp_prob_0.00005_base_500.0/ocrqa-wp_v2.0.0-lb.bloom
2025-12-06 01:19:44,128 ocrqa_create_bloom_filter.py:723 INFO: Writing metadata to build.d/fp_prob_0.00005_base_500.0/ocrqa-wp_v2.0.0-lb.bloom.json...
2025-12-06 01:19:44,128 ocrqa_create_bloom_filter.py:768 INFO: Metadata written to build.d/fp_prob_0.00005_base_500.0/ocrqa-wp_v2.0.0-lb.bloom.json
2025-12-06 01:19:44,128 ocrqa_create_bloom_filter.py:685 INFO: Writing lexicon to build.d/fp_prob_0.00005_base_500.0/ocrqa-wp_v2.0.0-lb.bloom.lexicon.txt...
2025-12-06 01:19:44,220 ocrqa_create_bloom_filter.py:689 INFO: Lexicon written to build.d/fp_prob_0.00005_base_500.0/ocrqa-wp_v2.0.0-lb.bloom.lexicon.txt (251359 words)
2025-12-06 01:19:44,268 ocrqa_create_bloom_filter.py:816 INFO: Diagnosis Results:
2025-12-06 01:19:44,268 ocrqa_create_bloom_filter.py:817 INFO:   - Excluded words in bloom filter: 0
2025-12-06 01:19:44,268 ocrqa_create_bloom_filter.py:818 INFO:   - Known words not in bloom filter: 0
2025-12-06 01:19:44,292 ocrqa_create_bloom_filter.py:825 INFO:   - Low-frequency words in bloom filter: 10
2025-12-06 01:19:44,292 ocrqa_create_bloom_filter.py:831 INFO:   - Proportion of excluded words in bloom filter: 0.00000000
2025-12-06 01:19:44,292 ocrqa_create_bloom_filter.py:837 INFO:   - Proportion of known words not in bloom filter: 0.00000000
2025-12-06 01:19:44,292 ocrqa_create_bloom_filter.py:845 INFO:   - Proportion of low-frequency words in bloom filter: 0.00005650