Kaggle-UWMGIT / Git / Diff of /README.md

Models:

Mathew-S/

Kaggle-UWMGIT

Downloads: 1

Diff of /README.md [4e96d3] .. [f1e01c]

Switch to unified view


### Introduction

Hello!

Below you can find a outline of how to reproduce my solution for the [UW-Madison GI Tract Image Segmentation | Kaggle](https://www.kaggle.com/competitions/uw-madison-gi-tract-image-segmentation/discussion/337197#1864282)

If you run into any trouble with the setup/code or have any questions please contact me at 273806108@qq.com



### Contents

```sh
preprocess.py: data preprocessing codes
inference.py: inferencing codes
other: necessary codes for `mmsegmentation` and `monai` toolboxes
```



### Hardware

```sh
Ubuntu 16.04 LTS (512 GB boot disk)
48 x Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
126 GB Memory
4 x NVIDIA Titan RTX
```

### Software

```sh
python==3.7.10
CUDA==10.2
cudnn==7.6.5
nvidia-drivers==440.4
(other refer to ./requirements.txt)
```

### Data setup

```sh
# DOWNLOAD DATA
kaggle competitions download -c uw-madison-gi-tract-image-segmentation

mkdir -p ./data/tract
mv uw-madison-gi-tract-image-segmentation.zip ./data/tract

cd ./data/tract
unzip uw-madison-gi-tract-image-segmentation.zip
cd ../..
```


The expected after unzip should be:

```sh
./data/tract
        ├── sample_submission.csv
        ├── test
        ├── train
        ├── train.csv
```
Install base requirements, `mmsegmentation` and `monai` toolboxes

```sh
# INSTALL PYTHON REQUIREMENTS
pip install -r requirements.txt
pip install "monai[ignite,skimage,nibabel]==0.8.1"
pip install mmcv-full==1.3.17 --force-reinstall -f https://download.openmmlab.com/mmcv/dist/cu102/torch1.10.0/index.html
pip install -v -e .
```

### Data preprocess

```sh
python data_preprocess.py
```

### Training

 NOTE: **make sure internet connection for public pretrained weights downloading**

```sh
mkdir -p saved_weights/cls saved_weights/seg saved_weights/3d

# DOWNLOAD PRETRAINED WEIGHTS
mkdir weights
cd weights
wget https://dl.fbaipublicfiles.com/convnext/ade20k/convnext_base_22k_224.pth
wget https://dl.fbaipublicfiles.com/convnext/ade20k/convnext_small_1k_224_ema.pth
cd ..

# TRAIN CLASSIFICATION MODELS
id=1
for config in $(find ./work_configs/tract/final_solution/classification_configs/cls*.py | sort); do
    ./tools/dist_train.sh $config 2
    last_work_dir=$(ls ./work_dirs/tract/ -rt | tail -n 1)
    last_weight=$(ls ./work_dirs/tract/$last_work_dir/*.pth -rt | tail -n 1)
    last_config=$(ls ./work_dirs/tract/$last_work_dir/*.py -rt | tail -n 1)
    mv ./work_dirs/tract/$last_work_dir/$last_weight ./saved_weights/cls/cls_${id}.pth
    mv ./work_dirs/tract/$last_work_dir/$last_config ./saved_weights/cls/cls_${id}.py
    id=$[id+1]
done

# TRAIN SEGMENTATION MODELS
id=1
for config in $(find ./work_configs/tract/final_solution/segmentation_configs/seg*.py | sort); do
    ./tools/dist_train.sh $config 2
    last_work_dir=$(ls ./work_dirs/tract/ -rt | tail -n 1)
    last_weight=$(ls ./work_dirs/tract/$last_work_dir/*.pth -rt | tail -n 1)
    last_config=$(ls ./work_dirs/tract/$last_work_dir/*.py -rt | tail -n 1)
    mv ./work_dirs/tract/$last_work_dir/$last_weight ./saved_weights/seg/seg_${id}.pth
    mv ./work_dirs/tract/$last_work_dir/$last_config ./saved_weights/seg/seg_${id}.py
    id=$[id+1]
done

# TRAIN 3D MODELS
cd ./monai
fold=-1
for n in (12 20 32); do
    mkdir -p  ./output/segres${n}_all/all
    python multilabel_train.py \
        -c segres${n}_all \
        -f $fold \
        > ./output/segres${n}_all/all/output.txt
        
    mkdir -p  ./output/segres${n}_all_round2/all
    python multilabel_train.py \
        -c segres${n}_all_round2 \
        -f $fold \
        -w ./output/segres${n}_all/all/last.pth \
        > ./output/segres${n}_all_round2/all/output.txt
    mv ./output/segres${n}_all_round2/all/last.pth ../saved_weights/3d/segres${n}.pth
done
cd ..
```

### Inferencing

```
python inference.py
```






	a/README.md		b/README.md
1	### Introduction	1	### Introduction
2		2
3	Hello!	3	Hello!
4		4
5	Below you can find a outline of how to reproduce my solution for the [UW-Madison GI Tract Image Segmentation \| Kaggle](https://www.kaggle.com/competitions/uw-madison-gi-tract-image-segmentation/discussion/337197#1864282)	5	Below you can find a outline of how to reproduce my solution for the [UW-Madison GI Tract Image Segmentation \| Kaggle](https://www.kaggle.com/competitions/uw-madison-gi-tract-image-segmentation/discussion/337197#1864282)
6		6
7	If you run into any trouble with the setup/code or have any questions please contact me at 273806108@qq.com	7	If you run into any trouble with the setup/code or have any questions please contact me at 273806108@qq.com
8		8
9		9
10		10
11	### Contents	11	### Contents
12		12
13	```sh	13	```sh
14	preprocess.py: data preprocessing codes	14	preprocess.py: data preprocessing codes
15	inference.py: inferencing codes	15	inference.py: inferencing codes
16	other: necessary codes for `mmsegmentation` and `monai` toolboxes	16	other: necessary codes for `mmsegmentation` and `monai` toolboxes
17	```	17	```
18		18
19		19
20		20
21	### Hardware	21	### Hardware
22		22
23	```sh	23	```sh
24	Ubuntu 16.04 LTS (512 GB boot disk)	24	Ubuntu 16.04 LTS (512 GB boot disk)
25	48 x Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz	25	48 x Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
26	126 GB Memory	26	126 GB Memory
27	4 x NVIDIA Titan RTX	27	4 x NVIDIA Titan RTX
28	```	28	```
29		29
30	### Software	30	### Software
31		31
32	```sh	32	```sh
33	python==3.7.10	33	python==3.7.10
34	CUDA==10.2	34	CUDA==10.2
35	cudnn==7.6.5	35	cudnn==7.6.5
36	nvidia-drivers==440.4	36	nvidia-drivers==440.4
37	(other refer to ./requirements.txt)	37	(other refer to ./requirements.txt)
38	```	38	```
39		39
40	### Data setup	40	### Data setup
41		41
42	```sh	42	```sh
43	# DOWNLOAD DATA	43	# DOWNLOAD DATA
44	kaggle competitions download -c uw-madison-gi-tract-image-segmentation	44	kaggle competitions download -c uw-madison-gi-tract-image-segmentation
45		45
46	mkdir -p ./data/tract	46	mkdir -p ./data/tract
47	mv uw-madison-gi-tract-image-segmentation.zip ./data/tract	47	mv uw-madison-gi-tract-image-segmentation.zip ./data/tract
48		48
49	cd ./data/tract	49	cd ./data/tract
50	unzip uw-madison-gi-tract-image-segmentation.zip	50	unzip uw-madison-gi-tract-image-segmentation.zip
51	cd ../..	51	cd ../..
52	```	52	```
53		53
54		54
55	The expected after unzip should be:	55	The expected after unzip should be:
56		56
57	```sh	57	```sh
58	./data/tract	58	./data/tract
59	├── sample_submission.csv	59	├── sample_submission.csv
60	├── test	60	├── test
61	├── train	61	├── train
62	├── train.csv	62	├── train.csv
63	```	63	```
64	Install base requirements, `mmsegmentation` and `monai` toolboxes	64	Install base requirements, `mmsegmentation` and `monai` toolboxes
65		65
66	```sh	66	```sh
67	# INSTALL PYTHON REQUIREMENTS	67	# INSTALL PYTHON REQUIREMENTS
68	pip install -r requirements.txt	68	pip install -r requirements.txt
69	pip install "monai[ignite,skimage,nibabel]==0.8.1"	69	pip install "monai[ignite,skimage,nibabel]==0.8.1"
70	pip install mmcv-full==1.3.17 --force-reinstall -f https://download.openmmlab.com/mmcv/dist/cu102/torch1.10.0/index.html	70	pip install mmcv-full==1.3.17 --force-reinstall -f https://download.openmmlab.com/mmcv/dist/cu102/torch1.10.0/index.html
71	pip install -v -e .	71	pip install -v -e .
72	```	72	```
73		73
74	### Data preprocess	74	### Data preprocess
75		75
76	```sh	76	```sh
77	python data_preprocess.py	77	python data_preprocess.py
78	```	78	```
79		79
80	### Training	80	### Training
81		81
82	> NOTE: make sure internet connection for public pretrained weights downloading	82	NOTE: make sure internet connection for public pretrained weights downloading
83		83
84	```sh	84	```sh
85	mkdir -p saved_weights/cls saved_weights/seg saved_weights/3d	85	mkdir -p saved_weights/cls saved_weights/seg saved_weights/3d
86		86
87	# DOWNLOAD PRETRAINED WEIGHTS	87	# DOWNLOAD PRETRAINED WEIGHTS
88	mkdir weights	88	mkdir weights
89	cd weights	89	cd weights
90	wget https://dl.fbaipublicfiles.com/convnext/ade20k/convnext_base_22k_224.pth	90	wget https://dl.fbaipublicfiles.com/convnext/ade20k/convnext_base_22k_224.pth
91	wget https://dl.fbaipublicfiles.com/convnext/ade20k/convnext_small_1k_224_ema.pth	91	wget https://dl.fbaipublicfiles.com/convnext/ade20k/convnext_small_1k_224_ema.pth
92	cd ..	92	cd ..
93		93
94	# TRAIN CLASSIFICATION MODELS	94	# TRAIN CLASSIFICATION MODELS
95	id=1	95	id=1
96	for config in $(find ./work_configs/tract/final_solution/classification_configs/cls*.py \| sort); do	96	for config in $(find ./work_configs/tract/final_solution/classification_configs/cls*.py \| sort); do
97	./tools/dist_train.sh $config 2	97	./tools/dist_train.sh $config 2
98	last_work_dir=$(ls ./work_dirs/tract/ -rt \| tail -n 1)	98	last_work_dir=$(ls ./work_dirs/tract/ -rt \| tail -n 1)
99	last_weight=$(ls ./work_dirs/tract/$last_work_dir/*.pth -rt \| tail -n 1)	99	last_weight=$(ls ./work_dirs/tract/$last_work_dir/*.pth -rt \| tail -n 1)
100	last_config=$(ls ./work_dirs/tract/$last_work_dir/*.py -rt \| tail -n 1)	100	last_config=$(ls ./work_dirs/tract/$last_work_dir/*.py -rt \| tail -n 1)
101	mv ./work_dirs/tract/$last_work_dir/$last_weight ./saved_weights/cls/cls_${id}.pth	101	mv ./work_dirs/tract/$last_work_dir/$last_weight ./saved_weights/cls/cls_${id}.pth
102	mv ./work_dirs/tract/$last_work_dir/$last_config ./saved_weights/cls/cls_${id}.py	102	mv ./work_dirs/tract/$last_work_dir/$last_config ./saved_weights/cls/cls_${id}.py
103	id=$[id+1]	103	id=$[id+1]
104	done	104	done
105		105
106	# TRAIN SEGMENTATION MODELS	106	# TRAIN SEGMENTATION MODELS
107	id=1	107	id=1
108	for config in $(find ./work_configs/tract/final_solution/segmentation_configs/seg*.py \| sort); do	108	for config in $(find ./work_configs/tract/final_solution/segmentation_configs/seg*.py \| sort); do
109	./tools/dist_train.sh $config 2	109	./tools/dist_train.sh $config 2
110	last_work_dir=$(ls ./work_dirs/tract/ -rt \| tail -n 1)	110	last_work_dir=$(ls ./work_dirs/tract/ -rt \| tail -n 1)
111	last_weight=$(ls ./work_dirs/tract/$last_work_dir/*.pth -rt \| tail -n 1)	111	last_weight=$(ls ./work_dirs/tract/$last_work_dir/*.pth -rt \| tail -n 1)
112	last_config=$(ls ./work_dirs/tract/$last_work_dir/*.py -rt \| tail -n 1)	112	last_config=$(ls ./work_dirs/tract/$last_work_dir/*.py -rt \| tail -n 1)
113	mv ./work_dirs/tract/$last_work_dir/$last_weight ./saved_weights/seg/seg_${id}.pth	113	mv ./work_dirs/tract/$last_work_dir/$last_weight ./saved_weights/seg/seg_${id}.pth
114	mv ./work_dirs/tract/$last_work_dir/$last_config ./saved_weights/seg/seg_${id}.py	114	mv ./work_dirs/tract/$last_work_dir/$last_config ./saved_weights/seg/seg_${id}.py
115	id=$[id+1]	115	id=$[id+1]
116	done	116	done
117		117
118	# TRAIN 3D MODELS	118	# TRAIN 3D MODELS
119	cd ./monai	119	cd ./monai
120	fold=-1	120	fold=-1
121	for n in (12 20 32); do	121	for n in (12 20 32); do
122	mkdir -p ./output/segres${n}_all/all	122	mkdir -p ./output/segres${n}_all/all
123	python multilabel_train.py \	123	python multilabel_train.py \
124	-c segres${n}_all \	124	-c segres${n}_all \
125	-f $fold \	125	-f $fold \
126	> ./output/segres${n}_all/all/output.txt	126	> ./output/segres${n}_all/all/output.txt
127		127
128	mkdir -p ./output/segres${n}_all_round2/all	128	mkdir -p ./output/segres${n}_all_round2/all
129	python multilabel_train.py \	129	python multilabel_train.py \
130	-c segres${n}_all_round2 \	130	-c segres${n}_all_round2 \
131	-f $fold \	131	-f $fold \
132	-w ./output/segres${n}_all/all/last.pth \	132	-w ./output/segres${n}_all/all/last.pth \
133	> ./output/segres${n}_all_round2/all/output.txt	133	> ./output/segres${n}_all_round2/all/output.txt
134	mv ./output/segres${n}_all_round2/all/last.pth ../saved_weights/3d/segres${n}.pth	134	mv ./output/segres${n}_all_round2/all/last.pth ../saved_weights/3d/segres${n}.pth
135	done	135	done
136	cd ..	136	cd ..
137	```	137	```
138		138
139	### Inferencing	139	### Inferencing
140		140
141	```	141	```
142	python inference.py	142	python inference.py
143	```	143	```
144		144
145		145
146		146
147		147
148		148