From nobody Thu Feb 12 21:59:29 2026 Received: from szxga03-in.huawei.com (szxga03-in.huawei.com [45.249.212.189]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2B48C13E8BD for ; Fri, 7 Jun 2024 04:29:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.189 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717734563; cv=none; b=Fy5GNJgEeAukwxNfy4983TojW0t0ivZFmhqUqe4F2kFAODxnPx/RwbeJZ8NgRAqP5Ej6sChQj5U/xMgaoaXkF0aCDH2CLxsd9V68C4Ik1l3nHtgxnmnHukGP3A/IXclf+C8IBZDq1U/qOBzt+HCjJ2aqnwpdKdgir0+f2CcB2U0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717734563; c=relaxed/simple; bh=XaHHnn2BmADtVuRCM50HTMTARh4wvX0PKd2u2NMrupM=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=qwMvdlmmPRFpK1++iVDqxVjKr45jqyxqWTSdVgzWO+bOvDdZVy3BS3tlvlNs1ovSJqetQB5nzdzYUVbkEb6mCamdaGEUhdxhBgdo5KVaiz9YQUfS6WcO5t+zikyDozL8M0DJuX4VGuPhpGn3HSs/q8CJfymZ5Gf68DgNOG/oJ/M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; arc=none smtp.client-ip=45.249.212.189 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Received: from mail.maildlp.com (unknown [172.19.88.105]) by szxga03-in.huawei.com (SkyGuard) with ESMTP id 4VwSn36p8qzPpnC; Fri, 7 Jun 2024 12:25:59 +0800 (CST) Received: from kwepemm600013.china.huawei.com (unknown [7.193.23.68]) by mail.maildlp.com (Postfix) with ESMTPS id 3D91F140413; Fri, 7 Jun 2024 12:29:04 +0800 (CST) Received: from huawei.com (10.175.104.67) by kwepemm600013.china.huawei.com (7.193.23.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 7 Jun 2024 12:27:39 +0800 From: Zhihao Cheng To: , , , , , CC: , Subject: [RFC PATCH mtd-utils 098/110] fsck.ubifs: Add README to describe fsck Date: Fri, 7 Jun 2024 12:26:03 +0800 Message-ID: <20240607042615.2069840-99-chengzhihao1@huawei.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240607042615.2069840-1-chengzhihao1@huawei.com> References: <20240607042615.2069840-1-chengzhihao1@huawei.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To kwepemm600013.china.huawei.com (7.193.23.68) Content-Type: text/plain; charset="utf-8" Add documents to describe fsck, which includes introductions, designment, advantage and limitations. Signed-off-by: Zhihao Cheng Signed-off-by: Zhang Yi --- ubifs-utils/fsck.ubifs/README.txt | 388 ++++++++++++++++++++++++++++++++++= ++++ 1 file changed, 388 insertions(+) create mode 100644 ubifs-utils/fsck.ubifs/README.txt diff --git a/ubifs-utils/fsck.ubifs/README.txt b/ubifs-utils/fsck.ubifs/REA= DME.txt new file mode 100644 index 00000000..99ad7c7c --- /dev/null +++ b/ubifs-utils/fsck.ubifs/README.txt @@ -0,0 +1,388 @@ + fsck.ubifs + =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + The fsck.ubifs can check and repair the UBIFS image on a given UBI volume= , it + could fix inconsistent UBIFS image(which is corrupted by hardware excepti= ons + or UBIFS realization bugs) and makes filesystem become consistent. + + + Manuals + ------- + There are four modes for fsck.ubifs: + 1. normal mode(no options): Check the filesystem, ask user whether to fix= the + problem as long as inconsistent data is found during fs checking. + 2. safe mode(-a option): Check and automatic safely repair the filesystem= , if + there are any data dropping operations needed by fixing, fsck will fai= l. + 3. danger mode(-y option): Answer 'yes' to all questions. There are two s= ub + modes: + a) Default submode(no options): Check and automatic repair the filesys= tem + according to TNC, data dropping will be reported. If TNC/master/log= is + corrupted, fsck will fail. + b) rebuild submode(-b option): Check and automatic forcibly repair the + filesystem, turns to rebuild filesystem if TNC/master/log is corrup= ted. + Always make fsck successful. + 4. check mode(-n option): Make no changes to the filesystem, only check t= he + filesystem. This mode doesn't check space, because unclean LEBs cannot= be + rewritten in read-only mode. + + The exit code returned by fsck.ubifs is compatible with FSCK(8), which is= the + sum of the following conditions: + 0 - No errors + 1 - File system errors corrected + 2 - System should be rebooted + 4 - File system errors left uncorrected + 8 - Operational error + 16 - Usage or syntax error + 32 - Fsck canceled by user request + 128 - Shared library error + + + Designment + ---------- + There are 2 working modes for fsck: rebuild mode and non-rebuild mode. Th= e main + idea is that construct all files by scanning the entire filesystem, then = check + the consistency of metadata(file meta information, space statistics, etc.) + according to the files. The file(xattr is treated as a file) is organized= as: + file tree(rbtree, inum indexed) + / \ + file1 file2 + / \ + file3 file4 + file { + inode node // each file has 1 inode node + dentry (sub rb_tree, sqnum indexed) + // '/' has no dentries, otherwise at least 1 dentry is required. + trun node // the newest one truncation node + data (sub rb_tree, block number indexed) + // Each file may have 0 or many data nodes + xattrs (sub rb_tree, inum indexed) + // Each file may have 0 or many xattr files + } + + Step 0. Both two modes need to read the superblock firstly, fsck fails if + superblock is corrupted, because fsck has no idea about the locat= ion + of each area(master, log, main, etc.) when the layout is lost. + + A. Rebuild mode: + Step 1. Scan nodes(inode node/dentry node/data node/truncation node) from= all + LEBs. + a) Corrupted LEBs(eg. garbage data, corrupted empty space) are dr= opped + during scanning. + b) Corrupted nodes(eg. incorrect crc, bad inode size, bad dentry = name + length, etc.) are dropped during scanning. + c) Valid inode nodes(nlink > 0) and dentry nodes(inum !=3D 0) are= put + into two valid trees(valid_inos & valid_dents) separately. + d) Deleted inode nodes (nlink is 0) and deleted dentry nodes(inum= is 0) + are put into two deleted trees(del_inos & del_dents) separatel= y. + e) Other nodes(data nodes/truncation node) are put into correspon= ding + file, if the file doesn't exist, insert a new file into the fi= le + tree. + Step 2. Traverse nodes from deleted trees, remove inode nodes and dentry = nodes + with smaller sqnum from valid trees. valid_inos - del_inos =3D le= ft_inos, + valid_dents - del_dents =3D left_dents. + This step handles the deleting case, for example, file A is delet= ed, + deleted inode node and deleted dentry node are written, if we ign= ore + the deleted nodes, file A can be recovered after rebuilding becau= se + undeleted inode node and undeleted dentry node can be scanned. Th= ere's + an exception, if deleted inode node and deleted dentry node are + reclaimed(by gc) after deletion, file A is recovered. So deleted = data + or files could be recovered by rebuild mode. + Step 3. Traverse left_inos and left_dents, insert inode node and dentry n= odes + into the corresponding file. + Step 4. Traverse all files, drop invalid files, move xattr files into the + corresponding host file's subtree. Invalid files such as: + a) File has no inode node or inode nlink is zero + b) Non-consistent file types between inode node and dentry nodes + c) File has no dentry nodes(excepts '/') + d) Encrypted file has no xattr information + e) Non regular file has data nodes + f) Directory/xattr file has more than one dentries + g) Xattr file has no host inode, or the host inode is a xattr + h) Non-xattr file's parent is not a directory + i) etc. + Step 5. Extract reachable directory entries tree. Make sure that all file= s can + be searched from '/', unreachable file is deleted. Since all xattr + files are attached to the corresponding host file, only non-xattr + files should be checked. Luckily, directory file only has one den= try, + the reachable checking of a dentry becomes easy. Traverse all + dentries for each file, check whether the dentry is reachable, if= not, + remove dentry from the file. If the file has no dentries, the fil= e is + unreachable. + Step 6. Correct the file information. Traverse all files and calculate + information(nlink, size, xattr_cnt, etc.) for each file just like + check_leaf(in linux kernel) does, correct the inode node based on= the + calculated information. + Step 7. Record used LEBs. Traverse all files'(including effective nodes f= rom + deletion trees in step 2) position, after this step fsck knows wh= ich + LEB is empty. + Step 8. Re-write data. Read data from LEB and write back data, make sure = that + all LEB is ended with empty data(0xFF). It will prevent failed gc + scanning in the next mounting. + Step 9. Build TNC. Construct TNC according to all files' nodes, just like= mkfs + does(refer to add_to_index in mkfs), then write TNC(refer to + write_index in mkfs) on flash. (If there are no files, create a n= ew + root dir file.) + Step 10.Build LPT. Construct LPT according to all nodes' position and len= gth, + just like mkfs does, then write LPT(refer to write_lpt) on flash. + Step 11.Clean up log area and orphan area. Log area and orphan area can be + erased. + Step 12.Write master node. Since all meta areas are ready, master node ca= n be + updated. + + B. Non-rebuild mode: + Step 1. Read master & init lpt. + a) Scan master nodes failed or master node is invalid (which is n= ot + caused by invalid space statistics), danger mode with rebuild_= fs and + normal mode with 'yes' answer will turn to rebuild mode, other= modes + will exit. Fsck cannot find the right TNC/LPT if the master no= de is + invalid, which affects subsequent steps, so this problem must = be + fixed. + b) Invalid space statistics in master node, set %FR_LPT_INCORRECT= for + for lpt status and ignore the error. + c) LPT node is corrupted, set %FR_LPT_CORRUPTED for lpt status and + ignore the error. + Step 2. Replay journal. + I. Scan log LEBs to get all buds. + a) Nodes in log LEBs are invalid/corrupted, danger mode with + rebuild_fs and normal mode with 'yes' answer will turn to r= ebuild + mode, other modes will exit. Corrupted log LEB could fail + ubifs_consolidate_log, which may lead to commit failure by = out of + space in the log area, so this problem must be fixed. + II. Scan bud LEBs to get all nodes. + a) Nodes in bud LEBs are invalid/corrupted, danger mode and n= ormal + mode with 'yes' answer will drop bud LEB and set + %FR_LPT_INCORRECT for lpt status, other modes will exit. + Corrupted LEB will make gc failed, so this problem must be + fixed. + III. Record isize into size tree according to data/truncation/inode + nodes. + IV. Apply nodes to TNC & LPT, update property for bud LEBs. + a) Corrupted/Invalid node searched from TNC, skip node and set + %FR_LPT_INCORRECT in lpt status for danger mode and normal= mode + with 'yes' answer, other modes will exit. The space statis= tics + depend on a valid TNC, so this problem must be fixed. + b) Corrupted/Invalid index node read from TNC, danger mode wi= th + rebuild_fs and normal mode with 'yes' answer will turn to + rebuild filesystem, other modes will exit. The space stati= stics + depend on a valid TNC, so this problem must be fixed. + c) Corrupted/Invalid lpt node, Set %FR_LPT_CORRUPTED for lpt = status + and ignore the error. + d) Incorrect LEB property: Set %FR_LPT_INCORRECT for lpt stat= us and + ignore the error. + e) If lpt status is not empty, skip updating lpt, because inc= orrect + LEB property could trigger assertion failure in ubifs_chan= ge_lp. + Step 3. Handle orphan nodes. + I. Scan orphan LEB to get all orphan nodes. + a) Corrupted/Invalid orphan node: danger mode and normal mode = with + 'yes' answer will drop orphan LEB, other modes will exit. + Corrupted orphan area could lead to mounting/committing fai= lure, + so this problem must be fixed. + II. Parse orphan node, find the original inode for each inum. + a) Corrupted/Invalid node searched from TNC, skip node for da= nger + mode and normal mode with 'yes' answer, other modes will e= xit. + b) Corrupted/Invalid index node read from TNC, danger mode wi= th + rebuild_fs and normal mode with 'yes' answer will turn to + rebuild filesystem, other modes will exit. The space stati= stics + depend on a valid TNC, so this problem must be fixed. + III. Remove inode for each inum, update TNC & LPT. + a) Corrupted/Invalid node searched from TNC, skip node for d= anger + mode and normal mode with 'yes' answer, other modes will = exit. + b) Corrupted/Invalid index node read from TNC, danger mode w= ith + rebuild_fs and normal mode with 'yes' answer will turn to + rebuild filesystem, other modes will exit. The space stat= istics + depend on a valid TNC, so this problem must be fixed. + c) Corrupted/Invalid lpt node, Set %FR_LPT_CORRUPTED for lpt + status and ignore the error. + d) Incorrect LEB property: Set %FR_LPT_INCORRECT for lpt sta= tus + and ignore the error. + e) If lpt status is not empty, skip updating lpt, because + incorrect LEB property could trigger assertion failure in + ubifs_change_lp. + Step 4. Consolidate log area. + a) Corrupted data in log LEBs, danger mode with rebuild_fs and no= rmal + mode with 'yes' answer will turn to rebuild filesystem, other = modes + will exit. It could make commit failed by out of space in log = area, + so this problem must be fixed. + Step 5. Recover isize. + I. Traverse size tree, lookup corresponding inode from TNC. + a) Corrupted/Invalid node searched from TNC, skip node for dan= ger + mode and normal mode with 'yes' answer, other modes will ex= it. + b) Corrupted/Invalid index node read from TNC, danger mode with + rebuild_fs and normal mode with 'yes' answer will turn to + rebuild filesystem, other modes will exit. The space statis= tics + depend on a valid TNC, so this problem must be fixed. + II. Update isize for inode. Keep in size tree for c= heck + mode, remove from the size tree and update inode + node in place for other modes. + Step 6. Traverse TNC and construct files. + I. Traverse TNC, check whether the leaf node is valid, remove inv= alid + nodes, construct file for valid node and insert the file into = the + file tree. + a) Corrupted/Invalid node searched from TNC, remove correspond= ing + TNC branch for danger mode and normal mode with 'yes' answe= r, + other modes will exit. The space statistics depend on a val= id + TNC, so this problem must be fixed. + b) Corrupted/Invalid index node read from TNC, danger mode with + rebuild_fs and normal mode with 'yes' answer will turn to + rebuild filesystem, other modes will exit. The space statis= tics + depend on a valid TNC, so this problem must be fixed. + II. Scan all LEBs(contain TNC) for non check mode(unclean LEBs ca= nnot + be fixed in read-only mode, so scanning may fail in check mod= e, + then space statistics won't be checked in check mode), remove= TNC + branch which points to corrupted LEB. + a) Corrupted data is found by scanning. If the current node is + index node, danger mode with rebuild_fs and normal mode wi= th + 'yes' answer will turn to rebuild filesystem, other modes = will + exit; If the current node is non-index node, danger mode a= nd + normal mode with 'yes' answer will remove all TNC branches= which + point to the corrupted LEB, other modes will exit. The spa= ce + statistics depend on valid LEB scanning, so this problem m= ust + be fixed. + b) LEB contains both index and non-index nodes, danger mode w= ith + rebuild_fs and normal mode with 'yes' answer will turn to + rebuild filesystem, other modes will exit. Invalid LEB wil= l make + gc failed, so this problem must be fixed. + Step 7. Update files' size for check mode. Update files' size according t= o the + size tree for check mode. + Step 8. Check and handle invalid files. Similar to rebuild mode, but the + methods of handling are different: + a) Move unattached(file has no dentries) regular file into discon= nected + list for safe mode, danger mode and normal mode with 'yes' ans= wer, + let subsequent steps to handle them with lost+found. Other mod= es + will exit. Disconnected file affects the result of calculated + information(which will be used in subsequent steps) for its' p= arent + file(eg. nlink, size), so this problem must be fixed. + b) Make file type be consistent between inode, detries and data n= odes + by deleting dentries or data nodes, for danger mode and normal= mode + with 'yes' answer, other modes will exit. + c) Delete file for other invalid cases(eg. file has no inode) in + danger mode and normal mode with 'yes' answer, other modes will + exit. + Step 9. Extract reachable directory entries tree. Similar to rebuild mode= , but + the methods of handling are different: + a) Remove unreachable dentry for danger mode and normal mode with= 'yes' + answer, other modes will exit. Unreachable dentry affects the + calculated information(which will be used in subsequent steps)= for + its' file(eg. nlink), so this problem must be fixed. + b) Delete unreachable non-regular file for danger mode and normal= mode + with 'yes' answer, other modes will exit. Unreachable file aff= ects + the calculated information(which will be used in subsequent st= eps) + for its' parent file(eg. nlink, size), so this problem must be + fixed. + c) Move unreachable regular file into disconnected list for safe = mode, + danger mode and normal mode with 'yes' answer, let subsequent = steps + to handle them with lost+found. Other modes will exit. Disconn= ected + file affects the calculated information(which will be used in + subsequent steps) for its' parent file(eg. nlink, size), so th= is + problem must be fixed. + Step 10.Correct the file information. Similar to rebuild mode, but the me= thods + of handling are different: + a) Correct the file information for safe mode, danger mode and no= rmal + mode with 'yes' answer, other modes will exit. Incorrect file + information affects the new creations(which will be used in ha= ndling + lost+found), so this problem must be fixed. + Step 11.Check whether the TNC is empty. Empty TNC is equal to corrupted T= NC, + which means that zero child count for root znode. If TNC is empty= (All + nodes are invalid and are deleted from TNC), turn to rebuild mode= for + danger mode with rebuild_fs and normal mode with 'yes' answer, ot= her + modes will exit. + Step 12.Check and correct the space statistics. + I. Exit for check mode, if %FR_LPT_CORRUPTED or %FR_LPT_INCORRECT= is + set in lpt status, the exit code should have %FSCK_UNCORRECTED. + II. Check lpt status, if %FR_LPT_CORRUPTED is set in lpt status, = normal + mode with 'no' answer will exit, other modes will rebuild lpt= . New + creations could be done in subsequent steps, which depends on + correct space statistics, so this problem must be fixed. + III. Traverse LPT nodes, check the correctness of nnode and pnode, + compare LEB scanning result with LEB properties. + a) LPT node is corrupted, normal mode with 'no' answer will = exit, + rebuild lpt for other modes. New creations could be done = in + subsequent steps, which depends on the correct space + statistics, so this problem must be fixed. + b) Incorrect nnode/pnode, normal mode with 'no' answer will = exit, + other other modes will correct the nnode/pnode. New creat= ions + could be done in subsequent steps, which depends on corre= ct + space statistics, so this problem must be fixed. + c) Inconsistent comparing result, normal mode with 'no' answ= er + will exit, other modes will correct the space statistics.= New + creations could be done in subsequent steps, which depend= s on + correct space statistics, so this problem must be fixed. + IV. Compare LPT area scanning result with lprops table informatio= n. + a) LPT area is corrupted, normal mode with 'no' answer will e= xit, + rebuild lpt for other modes. Commit could fail in doing LP= T gc + caused by scanning corrupted data, so this problem must be + fixed. + b) Inconsistent comparing result, normal mode with 'no' answer + will exit, other modes will correct the lprops table + information. Commit could fail in writing LPT with %ENOSPC + return code caused by incorrect space statistics in the LPT + area, so this problem must be fixed. + Step 13.Do commit, commit problem fixing modifications to disk. The index= size + checking depends on this step. + Step 14.Check and correct the index size. Check and correct the index siz= e by + traversing TNC just like dbg_check_idx_size does. This step shoul= d be + executed after first committing, because 'c->calc_idx_sz' can be + changed in 'ubifs_tnc_start_commit' and the initial value of + 'c->calc_idx_sz' read from the disk is untrusted. Correct the ind= ex + size for safe mode, danger mode and normal mode with 'yes' answer, + other modes will exit. New creations could be done in subsequent = steps, + which depends on the correct index size, so this problem must be = fixed. + Step 15.Check and create the root dir. Check whether the root dir exists, + create a new one if it is not found, for safe mode, danger mode a= nd + normal mode with 'yes' answer, other modes will exit. Mounting de= pends + on the root dir, so this problem must be fixed. + Step 16.Check and create the lost+found. + I. If the root dir is encrypted, set lost+found as invalid. Becau= se it + is impossible to check whether the lost+found exists in an enc= rypted + directory. + II. Search the lost+found under root dir. + a) Found a lost+found, lost+found is a non-encrypted director= y, set + lost+found as valid, otherwise set lost+found as invalid. + b) Not found the lost+found, create a new one. If creation is + failed by %ENOSPC, set lost+found as invalid. + Step 17.Handle each file from the disconnected list. + I. If lost+found is invalid, delete file for danger mode and norm= al + mode with 'yes' answer, other modes will skip and set the exit= code + with %FSCK_UNCORRECTED. + II. If lost+found is valid, link disconnected file under lost+found + directory with the name of the corresponding inode number + (INO__, index(starts from 0) is used to handle the + conflicted names). + a) Fails in handling conflicted file names, delete file for d= anger + mode and normal mode with 'yes' answer, other modes will s= kip + and set the exit code with %FSCK_UNCORRECTED. + b) Fails in linking caused by %ENOSPC, delete file for danger= mode + and normal mode with 'yes' answer, other modes will skip a= nd set + the exit code with %FSCK_UNCORRECTED. + Step 18.Do final commit, commit problem fixing modifications to disk and = clear + %UBIFS_MST_DIRTY flag for master node. + + + Advantage + --------- + 1. Can be used for any UBIFS image, fsck has nothing to do with kernel ve= rsion. + 2. Fsck is tolerant with power-cut, fsck will always succeed in a certain= mode + without changing mode even power-cut happens in checking and repairing= . In + other words, fsck won't let UBIFS image become worse in abnormal situa= tions. + 3. It is compatible with FSCK(8), the exit code returned by fsck.ubifs is= same + as FSCK, the command options used by fsck are supported in fsck.ubifs = too. + 4. The UBIFS image can be fixed as long as the super block is not corrupt= ed. + 5. Encrypted UBIFS image is supported, because dentry name and data conte= nt of + file are not necessary for fsck. + + + Limitations + ----------- + 1. UBIFS image file is not supported(Not like ext4). The UBIFS image file= is + not equal to UBI volume, empty LEBs are not included in image file, so= UBIFS + cannot allocate empty space when file recovering is needed. Another re= ason + is that atomic LEB changing is not supported by image file. + 2. Authenticated UBIFS image is not supported, UBIFS metadata(TNC/LPT) pa= rsing + depends on the authentication key which is not supported in fsck optio= ns. + + + Authors + ------- + Zhihao Cheng + Zhang Yi + Xiang Yang + Huang Xiaojia --=20 2.13.6