NSRLJP

Submitted by kazamiya on Sat, 04/24/2021 - 22:14

NSRLJP - complement to NSRL(http://www.nsrl.nist.gov/) hash library with Japanese edition and software.

Download

NSRLJP_202104.7z (SHA-256: ccf8cba3b4dbfd7c1252dbaaccbb661861794b205ad3a1d7884e785e12ca5d4d)
(Hashes: 4,142,667 / Filesize: 253,337,631 bytes)

License

You may use the DATASET freely for personal or commercial and NO WARRANTIES.

Motivation

National Software Reference Library (NSRL) provides Reference Data Set(RDS), which is a collection of digital signatures of known, traceable software applications. NSRLJP is a collection of a dataset which is widely used in Japan. It also includes Japanese edition of Microsoft Windows. NSRLJP is compliant with NSRL RDS data format so you can import NSRLJP into your favorite tools as well. Details of the data format, please refer to the following URL: 

Data Formats of the NSRL Reference Data Set (RDS) Distribution
http://www.nsrl.nist.gov/Documents/Data-Formats-of-the-NSRL-Reference-Data-Set-16.pdf

Catalog (NSRLJP_202104)

No. Name Count Comment (version, etc.)
1 Windows XP x64 10820  
2 Windows XP x86 16080 SP3
3 Windows 2003 R2 x64 16593 SP2
4 Windows 2003 R2 x86 14562 SP1
5 Windows Vista x64 22242 SP2
6 Windows Vista x86 15437 SP2
7 Windows 7 x64 40687 SP1
8 Windows 7 x86 15687 SP1
9 Windows 2008 x64 41249 SP2
10 Windows 2008 R2 x64 9335 SP1
11 Windows 8 x64 16145  
12 Windows 8 x86 13000  
13 Windows 8.1 x64 22199 Update
14 Windows 8.1 x86 19690 Update
15 Windows 2012 x64 45842  
16 Windows 2012 R2 x64 56929 Update
17 Windows 2016 64060  
18 Windows 10 x64 382617 1511, 1607, 1703, 1709, 1803, 1809, 1903, 1909, 2004, 20H2
19 Windows 10 x86 240783 1511, 1607, 1703, 1709, 1803, 1809, 1903, 1909, 2004, 20H2
20 Windows 2019 47623 Update (2019-03)
21 Office XP 4468 SP3
22 Office 2003 2210 SP1
23 Office 2007 2736 SP2
24 Office 2010 13680 SP2
25 Office 2013 11757 SP1
26 Office 2019 4036  
27 .NET Framework 2.x 491 v2.0
28 .NET Framework 3.x 872 v3.0, 3.5
29 .NET Framework 4.x 61429 v4.0, 4.5-4.5.2, 4.6-4.6.2, 4.7-4.7.2, 4.8
30 .NET5.x 1896  
31 Windows Update 2006 8765  
32 Windows Update 2007 17145 Except 2007-03
33 Windows Update 2008 19329 Except 2008-03
34 Windows Update 2009 36100 Except 2009-05
35 Windows Update 2010 59359 Except 2010-11
36 Windows Update 2011 68074  
37 Windows Update 2012 78872 Except 2012-09
38 Windows Update 2013 128707  
39 Windows Update 2014 425330  
40 Windows Update 2015 430520  
41 Windows Update 2016 174312 2016-01 ~ 08
42 Google Chrome 3489 73.0.3683.103-88.0.4324.104
43 Firefox 70894 0.8-87.0
44 Thunderbird 49973 0.4-78.9.1
45 Opera 5117 6.01-12.17
46 Adobe Reader 176609 6.x-11.0.23, 2015(-1500630527), DC(-2001320064), 2017(-1701130180), 2020(-2000130010)
47 Explzh 1105 6.06, 7.01-7.78, 8.17.4, 8.30-8.39
48 Lhaz 153 1.36, 2.1.3, 2.2.4, 2.4.0, 2.5.1, 3.3.0, 3.4.0, 3.5.1
49 Forefront Client Security 585  
50 Hidemaru Editor 1898 4.19-8.97
51 Hidemaru Mail 298 6.01-6.98
52 Sakura Editor 324 1.6.1.0-1.6.6.0, 2.0.4.0-2.2.0.1
53 Terapad 44 1.00-1.09
54 “Lhaca” 10 0.76, 0.97, 1.24
55 “Lhaplus” 26 1.71-1.74
56 WSUS Offline 1170074 2019-05-06, 2021-04-09
  Total 4142267  

The dataset is deduplicated based on MD5 and SHA-1. It means files have the same hash at various categories, only one record is registered into one of these categories.

Usage

We confirmed that the following tools support NSRLJP: 

  • X-ways Forensics
  • Autopsy 4.x
  • Magnet AXIOM 4.x

It's probably OSForensics, FTK, and md5deep also support.

Case Example

The following table shows the results that hashes with Windows OS are collated with NSRL, NSRLJP and both.

OS

Total number of files

(1) NSRL

(2) NSRLJP

(3) NSRL+NSRLJP

Windows 10 Pro (x64) JP

140370

10875

99662

99924

Windows 7 Ultimate SP1 (x64) JP

68995

16244

68059

68076

Windows Server 2008 Standard SP1 (x64) JP

64805

12350

53256

53547

Windows XP Professional SP3 (x86) JP

11050

5507

8170

8420

These OS are a clean install, we understand that NSRLJP includes most of hashes. In practical use, we should NSRL as well because it includes a wide variety of applications. Hash analysis is a traditional approach, but it will become increasingly important because the number of files in storages has increased year by year.

History

2021/04/24 (NSRLJP_202104 - 4,142,667 hash / 253,337,631 bytes)

Addition to Windows 10 (1909, 2004, 20H2), Windows 2019 (Update 2019/03), and Google Chrome.

2019/05/18 (NSRLJP_201905 - 3,718,659 hash / 228,100,979 bytes)

Addition to Windows 10 (1803, 1809, 1903) and Windows 2019.

2018/02/12 (NSRLJP_201802 - 2,993,931 hash / 180,917,699 bytes)

Addition to Windows 10 (1703, 1709) and Windows 2016, WSUS Offline Update for Windows Update.

2016/09/20 (NSRLJP_201609 - 2,309,928 hash / 158,073,246 bytes)

Addition to Windows 10 (1511, 1607), .NET Framework 4.6.1, 4.6.2 and Windows Update 2016.

2015/08/14 (NSRLJP_201508_rev2 - 1,659,348 hash / 114,136,669 bytes)

Excluded invalid record.

2015/08/11 (NSRLJP_201508)

Addition to Windows 10, .NET Framework 4.5.2, 4.6, Lhaca, Lhaplus and Windows Update 2015.

2014/08/10 (NSRLJP_201408 - 1,096,364 hash / 75,095,909 bytes)

Addition to Windows 8.1, 2012R2 Update, Office 2013 SP1, Office 2010 SP2, .NET Framework 3.5, 4.0, 4.5.1 and Windows Update 2014

2014/01/22 (NSRLJP_201401 - 532,923 hash / 38,245,480 bytes)

Addition to Windows 8.1, Windows 2008 R2 and Windows 2012.

2013/08/03 (NSRLJP_201308 - 401,211 hash / 28,657,266 bytes)

Addition to Windows Update, Adobe and Firefox.

2013/01/27 (NSRLJP_201301 - 284,419 hash / 20,673,998 bytes)

Addition to Windows 8/2012, Office XP/2003/2007/2010/2013 and .NET 2.0/3.0/4.5.

2011/09 (Initial release - 187,430 hash / 13,817,931 bytes)