2025-12-06 00:24:37,460 ocrqa_create_bloom_filter.py:282 INFO: Git version: 3bf39e8
2025-12-06 00:24:37,460 ocrqa_create_bloom_filter.py:284 INFO: Using normalization version: 2
2025-12-06 00:24:37,460 ocrqa_create_bloom_filter.py:286 INFO: Language-specific handling: de
2025-12-06 00:24:37,460 ocrqa_create_bloom_filter.py:288 INFO: Frequency threshold mode: adaptive
2025-12-06 00:24:37,460 ocrqa_create_bloom_filter.py:290 INFO: Adaptive parameters: base_freq=2500.0, exponent=2.0, min=2
2025-12-06 00:24:37,460 ocrqa_create_bloom_filter.py:1063 INFO: Namespace(input_files=['lex/de/realword_ocr_errors.nw.txt', 'lex/de/ocr_errors.nw.txt', 'lex/de/old_spelling.rw.txt', 'lex/de/modern_spelling.rw.txt', 'lex/de/realwords.rw.txt', 'lex/de/dewiki.unigram.freq.tsv.bz2'], bloom_path='build.d/fp_prob_0.00005_base_2500.0/ocrqa-wp_v2.0.0-de.bloom', fp_probability=5e-05, log_level='INFO', log_file='build.d/fp_prob_0.00005_base_2500.0/ocrqa-wp_v2.0.0-de.bloom.log', config=None, min_frequency=2, single_char_min_frequency=500, frequency_threshold_mode='adaptive', adaptive_base_freq=2500.0, adaptive_exponent=2.0, diagnose_bloom=True, git_version='3bf39e8', log_lexicon='build.d/fp_prob_0.00005_base_2500.0/ocrqa-wp_v2.0.0-de.bloom.lexicon.txt', version=2, language='de')
2025-12-06 00:24:37,461 ocrqa_create_bloom_filter.py:606 INFO: Starting Bloom Filter creation...
2025-12-06 00:24:37,461 ocrqa_create_bloom_filter.py:469 INFO: Processing nonword file: lex/de/realword_ocr_errors.nw.txt
2025-12-06 00:24:37,462 ocrqa_create_bloom_filter.py:495 INFO: Exclusion list built: 51 words will be filtered from bloom filter
2025-12-06 00:24:37,462 ocrqa_create_bloom_filter.py:469 INFO: Processing nonword file: lex/de/ocr_errors.nw.txt
2025-12-06 00:24:37,463 ocrqa_create_bloom_filter.py:495 INFO: Exclusion list built: 314 words will be filtered from bloom filter
2025-12-06 00:24:37,463 ocrqa_create_bloom_filter.py:528 INFO: Processing real-word file: lex/de/old_spelling.rw.txt
2025-12-06 00:24:37,606 ocrqa_create_bloom_filter.py:528 INFO: Processing real-word file: lex/de/modern_spelling.rw.txt
2025-12-06 00:24:37,728 ocrqa_create_bloom_filter.py:528 INFO: Processing real-word file: lex/de/realwords.rw.txt
2025-12-06 00:24:37,728 ocrqa_create_bloom_filter.py:385 INFO: Processing frequency file: lex/de/dewiki.unigram.freq.tsv.bz2
2025-12-06 00:24:37,741 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 3: 277
2025-12-06 00:24:37,742 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 4: 156
2025-12-06 00:24:37,742 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 2: 625
2025-12-06 00:24:37,742 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 5: 100
2025-12-06 00:24:37,742 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 6: 69
2025-12-06 00:24:37,742 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 1: 2500
2025-12-06 00:24:37,742 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 8: 39
2025-12-06 00:24:37,742 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 7: 51
2025-12-06 00:24:37,742 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 9: 30
2025-12-06 00:24:37,742 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 11: 20
2025-12-06 00:24:37,742 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 10: 25
2025-12-06 00:24:37,742 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 12: 17
2025-12-06 00:24:37,743 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 13: 14
2025-12-06 00:24:37,743 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 14: 12
2025-12-06 00:24:37,743 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 15: 11
2025-12-06 00:24:37,745 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 16: 9
2025-12-06 00:24:37,745 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 17: 8
2025-12-06 00:24:37,746 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 18: 7
2025-12-06 00:24:37,747 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 19: 6
2025-12-06 00:24:37,752 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 21: 5
2025-12-06 00:24:37,753 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 20: 6
2025-12-06 00:24:37,756 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 23: 4
2025-12-06 00:24:37,763 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 22: 5
2025-12-06 00:24:37,765 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 25: 4
2025-12-06 00:24:37,769 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 24: 4
2025-12-06 00:24:37,779 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 30: 2
2025-12-06 00:24:37,797 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 27: 3
2025-12-06 00:24:37,802 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 34: 2
2025-12-06 00:24:37,811 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 26: 3
2025-12-06 00:24:37,820 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 28: 3
2025-12-06 00:24:37,825 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 36: 2
2025-12-06 00:24:37,835 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 31: 2
2025-12-06 00:24:37,844 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 29: 2
2025-12-06 00:24:37,950 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 42: 2
2025-12-06 00:24:37,952 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 32: 2
2025-12-06 00:24:38,033 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 33: 2
2025-12-06 00:24:38,051 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 40: 2
2025-12-06 00:24:38,279 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 35: 2
2025-12-06 00:24:38,398 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 37: 2
2025-12-06 00:24:38,603 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 38: 2
2025-12-06 00:24:38,704 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 44: 2
2025-12-06 00:24:39,187 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 39: 2
2025-12-06 00:24:39,415 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 41: 2
2025-12-06 00:24:39,621 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 48: 2
2025-12-06 00:24:39,931 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 54: 2
2025-12-06 00:24:40,430 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 45: 2
2025-12-06 00:24:40,442 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 43: 2
2025-12-06 00:24:40,535 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 46: 2
2025-12-06 00:24:41,374 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 55: 2
2025-12-06 00:24:43,184 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 64: 2
2025-12-06 00:24:43,292 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 47: 2
2025-12-06 00:24:43,950 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 58: 2
2025-12-06 00:24:44,461 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 52: 2
2025-12-06 00:24:46,581 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 49: 2
2025-12-06 00:24:47,153 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 51: 2
2025-12-06 00:24:47,290 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 50: 2
2025-12-06 00:24:47,830 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 53: 2
2025-12-06 00:24:48,308 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 70: 2
2025-12-06 00:24:48,480 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 60: 2
2025-12-06 00:24:49,370 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 56: 2
2025-12-06 00:24:49,893 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 85: 2
2025-12-06 00:24:50,535 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 63: 2
2025-12-06 00:24:52,633 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 97: 2
2025-12-06 00:24:52,691 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 108: 2
2025-12-06 00:24:53,037 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 90: 2
2025-12-06 00:24:53,653 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 57: 2
2025-12-06 00:24:54,106 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 61: 2
2025-12-06 00:24:54,552 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 73: 2
2025-12-06 00:24:54,903 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 65: 2
2025-12-06 00:24:56,208 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 59: 2
2025-12-06 00:24:57,101 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 68: 2
2025-12-06 00:24:57,477 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 80: 2
2025-12-06 00:24:58,148 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 62: 2
2025-12-06 00:24:58,494 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 95: 2
2025-12-06 00:24:58,637 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 123: 2
2025-12-06 00:24:58,776 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 66: 2
2025-12-06 00:24:58,872 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 69: 2
2025-12-06 00:24:59,305 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 87: 2
2025-12-06 00:24:59,478 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 98: 2
2025-12-06 00:24:59,577 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 74: 2
2025-12-06 00:25:00,281 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 72: 2
2025-12-06 00:25:01,853 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 67: 2
2025-12-06 00:25:02,350 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 377: 2
2025-12-06 00:25:03,130 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 77: 2
2025-12-06 00:25:03,913 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 250: 2
2025-12-06 00:25:04,029 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 81: 2
2025-12-06 00:25:04,409 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 76: 2
2025-12-06 00:25:05,589 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 92: 2
2025-12-06 00:25:05,918 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 79: 2
2025-12-06 00:25:06,575 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 107: 2
2025-12-06 00:25:06,857 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 71: 2
2025-12-06 00:25:07,075 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 106: 2
2025-12-06 00:25:07,282 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 75: 2
2025-12-06 00:25:07,752 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 131: 2
2025-12-06 00:25:08,468 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 135: 2
2025-12-06 00:25:10,555 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 94: 2
2025-12-06 00:25:11,187 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 115: 2
2025-12-06 00:25:11,484 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 83: 2
2025-12-06 00:25:11,508 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 138: 2
2025-12-06 00:25:15,377 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 118: 2
2025-12-06 00:25:16,625 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 100: 2
2025-12-06 00:25:16,864 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 142: 2
2025-12-06 00:25:17,138 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 152: 2
2025-12-06 00:25:19,204 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 136: 2
2025-12-06 00:25:19,332 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 89: 2
2025-12-06 00:25:23,431 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 116: 2
2025-12-06 00:25:23,985 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 103: 2
2025-12-06 00:25:24,125 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 93: 2
2025-12-06 00:25:24,457 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 86: 2
2025-12-06 00:25:24,579 ocrqa_create_bloom_filter.py:360 INFO: Frequency threshold for word length 99: 2
2025-12-06 00:25:25,178 ocrqa_create_bloom_filter.py:623 INFO: Low-frequency exclusions before cleanup: 8111926 tokens
2025-12-06 00:25:25,243 ocrqa_create_bloom_filter.py:631 INFO: Low-frequency exclusions after cleanup: 8029879 tokens
2025-12-06 00:25:25,243 ocrqa_create_bloom_filter.py:635 INFO: Lexical processing complete.
2025-12-06 00:25:25,243 ocrqa_create_bloom_filter.py:638 INFO: Processing Statistics:
2025-12-06 00:25:25,243 ocrqa_create_bloom_filter.py:640 INFO:   freq_words_filtered: 8,337,944
2025-12-06 00:25:25,243 ocrqa_create_bloom_filter.py:640 INFO:   freq_words_read: 9,199,714
2025-12-06 00:25:25,243 ocrqa_create_bloom_filter.py:640 INFO:   low_freq_excluded: 8,029,879
2025-12-06 00:25:25,243 ocrqa_create_bloom_filter.py:640 INFO:   nonwords_count: 315
2025-12-06 00:25:25,243 ocrqa_create_bloom_filter.py:640 INFO:   nonwords_read: 315
2025-12-06 00:25:25,243 ocrqa_create_bloom_filter.py:640 INFO:   realwords_accepted: 95,331
2025-12-06 00:25:25,243 ocrqa_create_bloom_filter.py:640 INFO:   realwords_nonwords_filtered: 6
2025-12-06 00:25:25,243 ocrqa_create_bloom_filter.py:640 INFO:   realwords_read: 95,337
2025-12-06 00:25:25,243 ocrqa_create_bloom_filter.py:640 INFO:   single_char_words_filtered: 7,885
2025-12-06 00:25:25,243 ocrqa_create_bloom_filter.py:640 INFO:   subwords_accepted: 868,242
2025-12-06 00:25:25,243 ocrqa_create_bloom_filter.py:640 INFO:   subwords_filtered: 281
2025-12-06 00:25:25,243 ocrqa_create_bloom_filter.py:640 INFO:   words_accepted: 853,885
2025-12-06 00:25:25,243 ocrqa_create_bloom_filter.py:644 INFO: Creating bloom filter with 890,956 unique words/tokens
2025-12-06 00:25:25,243 ocrqa_create_bloom_filter.py:647 INFO: Target false positive probability: 5e-05
2025-12-06 00:25:25,452 ocrqa_create_bloom_filter.py:652 INFO: Bloom filter created successfully: build.d/fp_prob_0.00005_base_2500.0/ocrqa-wp_v2.0.0-de.bloom
2025-12-06 00:25:25,452 ocrqa_create_bloom_filter.py:723 INFO: Writing metadata to build.d/fp_prob_0.00005_base_2500.0/ocrqa-wp_v2.0.0-de.bloom.json...
2025-12-06 00:25:25,452 ocrqa_create_bloom_filter.py:768 INFO: Metadata written to build.d/fp_prob_0.00005_base_2500.0/ocrqa-wp_v2.0.0-de.bloom.json
2025-12-06 00:25:25,452 ocrqa_create_bloom_filter.py:685 INFO: Writing lexicon to build.d/fp_prob_0.00005_base_2500.0/ocrqa-wp_v2.0.0-de.bloom.lexicon.txt...
2025-12-06 00:25:25,889 ocrqa_create_bloom_filter.py:689 INFO: Lexicon written to build.d/fp_prob_0.00005_base_2500.0/ocrqa-wp_v2.0.0-de.bloom.lexicon.txt (890956 words)
2025-12-06 00:25:26,035 ocrqa_create_bloom_filter.py:816 INFO: Diagnosis Results:
2025-12-06 00:25:26,035 ocrqa_create_bloom_filter.py:817 INFO:   - Excluded words in bloom filter: 0
2025-12-06 00:25:26,035 ocrqa_create_bloom_filter.py:818 INFO:   - Known words not in bloom filter: 0
2025-12-06 00:25:26,948 ocrqa_create_bloom_filter.py:825 INFO:   - Low-frequency words in bloom filter: 374
2025-12-06 00:25:26,948 ocrqa_create_bloom_filter.py:831 INFO:   - Proportion of excluded words in bloom filter: 0.00000000
2025-12-06 00:25:26,948 ocrqa_create_bloom_filter.py:837 INFO:   - Proportion of known words not in bloom filter: 0.00000000
2025-12-06 00:25:26,948 ocrqa_create_bloom_filter.py:845 INFO:   - Proportion of low-frequency words in bloom filter: 0.00004658