-
Notifications
You must be signed in to change notification settings - Fork 0
/
feed.xml
300 lines (211 loc) · 44.6 KB
/
feed.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="4.3.1">Jekyll</generator><link href="https://nadav.amit.zone/feed.xml" rel="self" type="application/atom+xml" /><link href="https://nadav.amit.zone/" rel="alternate" type="text/html" /><updated>2023-12-26T12:28:36+02:00</updated><id>https://nadav.amit.zone/feed.xml</id><title type="html">Nadav Amit</title><subtitle>Fighting Operating Systems</subtitle><entry><title type="html">How new-lines affect the Linux kernel performance</title><link href="https://nadav.amit.zone/linux/2018/10/11/newline.html" rel="alternate" type="text/html" title="How new-lines affect the Linux kernel performance" /><published>2018-10-11T00:52:01+03:00</published><updated>2018-10-11T00:52:01+03:00</updated><id>https://nadav.amit.zone/linux/2018/10/11/newline</id><content type="html" xml:base="https://nadav.amit.zone/linux/2018/10/11/newline.html"><![CDATA[<p>The Linux kernel strives to be fast and efficient. As it is written mostly in C, it can mostly control how the generated machine code looks. Nevertheless, as the kernel code is compiled into machine code, the compiler optimizes the generated code to improve its performance. The kernel code, however, employs uncommon coding techniques, which can fail code optimizations. In this blog-post, I would share my experience in analyzing the reasons for poor code inlining of the kernel code. Although the performance improvement are not significant in most cases, understanding these issues are valuable in preventing them from becoming larger. New-lines, as promised, will be one of the reasons, though not the only one.</p>
<h2 id="new-lines-in-inline-assembly">New lines in inline assembly</h2>
<p>One fine day, I encountered a strange phenomenon: minor changes I performed in the Linux source code, caused small but noticeable performance degradation. As I expected these changes to actually improve performance, I decided to disassemble the functions which I changed. To my surprise, I realized that my change caused functions that were previously inlined, not to be inlined anymore. The decision not to inline these functions seem dubious as they were short.</p>
<p>I decided to further investigate this issue and to check whether it affects other parts of the kernel. Arguably, it is rather hard to say whether a function should be inlined, so some sort of indication of bad inlining decisions is needed. C functions that are declared with the inline keyword are not bound to be inlined by the compiler, so having a non-inlined function that is marked with the inline keyword is not an indication by itself for bad inlining decision.</p>
<p>Arguably, there are two simple heuristics to find functions which were suspiciously not inlined for the wrong reason. One heuristic is to look for short (binary-wise) functions by looking at the static symbols. A second heuristic is to look for functions which appear in multiple translation units (objects), as this might indicate they were declared as inline but were eventually not inlined, and that they are in common use. In both cases, there may be valid reasons for the compiler not to inline functions even if they are short, for example if they are used as a value for a function pointer. However, they can give an indication if something is “very wrong” in how inlining is performed, or more correctly, ignored.</p>
<p>In practice, I used both heuristics, but in this post I will only use the second one to check whether inlining decisions seem dubious. To do so I rebuild the kernel, using the localyesconfig make target to incorporate the modules into the core. I ensure the “kernel hacking” features in the config are off, as those tend to blow the size of the code and rightfully cause functions not to be inlined. I then looked for static function which had the most instances in the built kernel:</p>
<figure class="highlight"><pre><code class="language-console" data-lang="console"><span class="gp">$</span><span class="w"> </span>nm <span class="nt">--print-size</span> ./vmlinux | <span class="nb">grep</span> <span class="s1">' t '</span> | <span class="nb">cut</span> <span class="nt">-d</span><span class="s1">' '</span> <span class="nt">-f2-</span> | <span class="nb">sort</span> | <span class="nb">uniq</span> <span class="nt">-c</span> <span class="se">\</span>
<span class="go"> | grep -v '^ 1' | sort -n -r | head -n 5
Instances Size Function Name
36 0000000000000019 t copy_overflow
8 000000000000012f t jhash
8 000000000000000d t arch_local_save_flags
7 0000000000000017 t dst_output
6 000000000000004e t put_page</span></code></pre></figure>
<p>As seen, the results are suspicious. As mentioned before, in some cases there are good reasons for functions not to be inlined. jhash() is a big function (303 bytes) so it is reasonable for it is not to be inlined. dst_output() address is used as a function pointer, which causes it not to be inlined. Yet the other functions seem to be great candidates for inlining, and it is not clear why they are not inlined. Let’s look at the source code of copy_overflow(), which has many instances in the binary:</p>
<figure class="highlight"><pre><code class="language-c" data-lang="c"><span class="k">static</span> <span class="kr">inline</span> <span class="kt">void</span> <span class="nf">copy_overflow</span><span class="p">(</span><span class="kt">int</span> <span class="n">size</span><span class="p">,</span> <span class="kt">unsigned</span> <span class="kt">long</span> <span class="n">count</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">WARN</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s">"Buffer overflow detected (%d < %lu)!</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">size</span><span class="p">,</span> <span class="n">count</span><span class="p">);</span>
<span class="p">}</span></code></pre></figure>
<p>Will the disassembly tell us anything?</p>
<figure class="highlight"><pre><code class="language-objdump" data-lang="objdump"><table class="rouge-table"><tbody><tr><td class="gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
</pre></td><td class="code"><pre>0xffffffff819315e0 <+0>: push %rbp
0xffffffff819315e1 <+1>: mov %rsi,%rdx
0xffffffff819315e4 <+4>: mov %edi,%esi
0xffffffff819315e6 <+6>: mov $0xffffffff820bc4b8,%rdi
0xffffffff819315ed <+13>: mov %rsp,%rbp
0xffffffff819315f0 <+16>: callq 0xffffffff81089b70 <__warn_printk>
0xffffffff819315f5 <+21>: ud2
0xffffffff819315f7 <+23>: pop %rbp
0xffffffff819315f8 <+24>: retq
</pre></td></tr></tbody></table></code></pre></figure>
<p>Apparently not. Notice that out of the 9 assembly instructions that are shown above, 6 deal with the function entry and exit - for example, updating the frame pointer, and only the three (in lines 4, 6 and 7) are really needed.</p>
<p>To understand the problem, we must dig deeper and look at the warning mechanism in Linux. In x86, this mechanism shares the same infrastructure with the bug reporting mechanism. When a bug or a warning are triggered, the kernel prints the filename and the line number in the source-code that triggered the bug, which can then used to analyze the root-cause of the bug. A naive implementation, however, would cause the code-cache to be polluted with the this information as well as the function call to the function that prints the error message, consequently causing performance degradation.</p>
<p>Linux therefore uses a different scheme by setting an exception triggering instruction (<a href="https://c9x.me/x86/html/file_module_x86_id_318.html">ud2</a> on x86) and saving the warning information in a bug table that is set in a different section in the executable. Once a warning is triggered using the WARN() macro, an exception is triggered and the exception handler looks for the warning information - the source-code filename and line - in the table.</p>
<p>Inline assembly is used to save this information in <a href="https://elixir.bootlin.com/linux/v4.17/source/arch/x86/include/asm/bug.h#L33">_BUG_FLAGS()</a>. Here is its code after some simplifications to ease readability:</p>
<figure class="highlight"><pre><code class="language-c" data-lang="c"><span class="n">asm</span> <span class="nf">volatile</span><span class="p">(</span><span class="s">"1: ud2</span><span class="se">\n</span><span class="s">"</span>
<span class="s">".pushsection __bug_table,</span><span class="se">\"</span><span class="s">aw</span><span class="se">\"\n</span><span class="s">"</span>
<span class="s">"2: .long 1b - 2b</span><span class="se">\n</span><span class="s">"</span> <span class="cm">/* bug_entry::bug_addr */</span>
<span class="s">" .long %c0 - 2b</span><span class="se">\n</span><span class="s">"</span> <span class="cm">/* bug_entry::file */</span>
<span class="s">" .word %c1</span><span class="se">\n</span><span class="s">"</span> <span class="cm">/* bug_entry::line */</span>
<span class="s">" .word %c2</span><span class="se">\n</span><span class="s">"</span> <span class="cm">/* bug_entry::flags */</span>
<span class="s">" .org 2b+%c3</span><span class="se">\n</span><span class="s">"</span>
<span class="s">".popsection"</span>
<span class="o">:</span> <span class="o">:</span> <span class="s">"i"</span> <span class="p">(</span><span class="n">__FILE__</span><span class="p">),</span> <span class="s">"i"</span> <span class="p">(</span><span class="n">__LINE__</span><span class="p">),</span>
<span class="s">"i"</span> <span class="p">(</span><span class="n">flags</span><span class="p">),</span>
<span class="s">"i"</span> <span class="p">(</span><span class="k">sizeof</span><span class="p">(</span><span class="k">struct</span> <span class="n">bug_entry</span><span class="p">)));</span></code></pre></figure>
<p>Ignoring the assembly shenanigans that this code uses, we can see that in practice it generates a single ud2 instruction. However, the compiler considers this code to be “big” and consequently oftentimes does not inline functions that use WARN() or similar functions.</p>
<p>The reason turns to be the newline characters (marked as ‘\n’ above). The kernel compiler, GCC, is unaware to the code size that will be generated by the inline assembly. It therefore tries to estimate its size based on newline characters and statement separators (‘;’ on x86). In GCC, we can see the code that performs this estimation in the estimate_num_insns() function:</p>
<figure class="highlight"><pre><code class="language-c" data-lang="c"><span class="kt">int</span> <span class="nf">estimate_num_insns</span> <span class="p">(</span><span class="n">gimple</span> <span class="o">*</span><span class="n">stmt</span><span class="p">,</span> <span class="n">eni_weights</span> <span class="o">*</span><span class="n">weights</span><span class="p">)</span>
<span class="p">{</span>
<span class="c1">// ...</span>
<span class="k">case</span> <span class="n">GIMPLE_ASM</span><span class="p">:</span>
<span class="p">{</span>
<span class="kt">int</span> <span class="n">count</span> <span class="o">=</span> <span class="n">asm_str_count</span> <span class="p">(</span><span class="n">gimple_asm_string</span> <span class="p">(</span><span class="n">as_a</span> <span class="o"><</span><span class="n">gasm</span> <span class="o">*></span> <span class="p">(</span><span class="n">stmt</span><span class="p">)));</span>
<span class="cm">/* 1000 means infinity. This avoids overflows later
with very long asm statements. */</span>
<span class="k">if</span> <span class="p">(</span><span class="n">count</span> <span class="o">></span> <span class="mi">1000</span><span class="p">)</span>
<span class="n">count</span> <span class="o">=</span> <span class="mi">1000</span><span class="p">;</span>
<span class="k">return</span> <span class="n">count</span><span class="p">;</span>
<span class="p">}</span>
<span class="c1">// ...</span>
<span class="p">}</span></code></pre></figure>
<p>Note that this pattern, of saving data using inline assembly, is not limited to bugs and warnings. The kernel uses it for many additional purposes: exception tables, that gracefully handle an exception that is triggered inside the kernel; alternative instructions table, that tailors the kernel on boot-time to the specific CPU architecture extensions that are supported; annotations that are used for stack metadata validation by objtool and so on.</p>
<p>Before we get to solving this problem, a question needs to be raised: is the current behavior flawed at all? Eventually, the size of the kernel will increase if functions that use WARN(), for example, will be inlined. This increase in size can cause the kernel image to be bigger, and since the Linux kernel cannot be paged out, will also increase memory consumption. However, the main reason that the compiler strives to avoid inflation of the code size is to avoid pressure on the instruction cache, whose impact may offset inlining benefits. Moreover, the heuristics of other compiler optimizations (e.g., loop optimizations) depend on the size of the code.</p>
<p>Solving the problem is not trivial. Ideally, GCC would have used an integrated assembler, similarly to <a href="https://www.embecosm.com/appnotes/ean10/ean10-howto-llvmas-1.0.html#idp109760">LLVM</a>, which would give better estimation of the generated code size of inline assembly. Experimentally, LLVM seems to make the right inlining decisions and is not affected by new-lines or data that is set in other sections of the executable. Interestingly, it appears to do so even when the integrated assembler is not used for assembly. GCC, however, uses the GNU assembler after the code is compiled, which prevents it from getting a correct estimation of the code size.</p>
<p>Alternatively, the problem could have been solved by overriding GCC’s code size estimation through a directive or a built-in function. However, looking at GCC code does not reveal a direct or indirect way to achieve this goal.</p>
<p>One may think that using the always_inline function attribute to force the compiler to inline functions would solve the problem. It appears that some have encountered the problem of poor inlining decisions in the past, without understanding the root-cause and used this solution. However, this solution has several drawbacks. First, it is hard to make and maintain these annotations. Second, this solution does not address other code optimizations the rely on code-size estimation. Third, the kernel uses various configurations and supports multiple CPU architectures, which may require a certain function to be inlined in some setups and not inlined in other. Finally, and most importantly, using always_inline can just push the problem upwards to calling functions, as we will later see.</p>
<p>Therefore, a more systematic solution is needed. The solution comes in the form of assembly macros that are set to hold the long assembly code, and use a single line inside the inline assembly that calls the macro. This solution does not only improve the generated machine code, but makes the assembly code more readable, as it prevents various quirks that are required in inline assembly, for example new-line characters. Moreover, in certain cases this change allows to consolidate the currently separate implementations that are used in C and assembly, which eases code maintenance.</p>
<p>Addressing the issue shows a performance improvement of tens of cycles for certain system calls, which are indeed not too notable. After addressing these issues, we see copy_overflow() and other functions disappear from the commonly non-inlined inline functions list.</p>
<figure class="highlight"><pre><code class="language-console" data-lang="console"><span class="gp">$</span><span class="w"> </span>nm <span class="nt">--print-size</span> ./vmlinux | <span class="nb">grep</span> <span class="s1">' t '</span> | <span class="nb">cut</span> <span class="nt">-d</span><span class="s1">' '</span> <span class="nt">-f2-</span> | <span class="nb">sort</span> | <span class="nb">uniq</span> <span class="nt">-c</span> <span class="se">\</span>
<span class="go"> | grep -v '^ 1' | sort -n -r | head -n 5
Instances Size Function Name
9 000000000000012f t jhash
8 0000000000000011 t kzalloc
7 0000000000000017 t dst_output
5 000000000000002f t acpi_os_allocate_zeroed
5 0000000000000029 t acpi_os_allocate</span></code></pre></figure>
<p>However, we got some new ones. Lets try to understand where do they come from.</p>
<h2 id="constant-computations-and-inlining">Constant computations and inlining</h2>
<p>As shown, kzalloc() is not always inlined, although its code is very simple.</p>
<figure class="highlight"><pre><code class="language-c" data-lang="c"><span class="k">static</span> <span class="kr">inline</span> <span class="kt">void</span> <span class="o">*</span><span class="nf">kzalloc</span><span class="p">(</span><span class="kt">size_t</span> <span class="n">size</span><span class="p">,</span> <span class="n">gfp_t</span> <span class="n">flags</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">kmalloc</span><span class="p">(</span><span class="n">size</span><span class="p">,</span> <span class="n">flags</span> <span class="o">|</span> <span class="n">__GFP_ZERO</span><span class="p">);</span>
<span class="p">}</span></code></pre></figure>
<p>The assembly, again does not provide any answers as to why it is not inlined:</p>
<figure class="highlight"><pre><code class="language-objdump" data-lang="objdump">0xffffffff817929e0 <+0>: push %rbp
0xffffffff817929e1 <+1>: mov $0x14080c0,%esi
0xffffffff817929e6 <+6>: mov %rsp,%rbp
0xffffffff817929e9 <+9>: callq 0xffffffff8125d590 <__kmalloc>
0xffffffff817929ee <+14>: pop %rbp
0xffffffff817929ef <+15>: retq </code></pre></figure>
<p>The answer to our question lies in kmalloc(), which is called by kzalloc() and is considered to have many instructions by GCC heuristics. kmalloc() is inlined since it is marked with the always_inline attribute, but its estimated instruction count is then attributed to the calling function, kzalloc() in this case. This result exemplifies why the use of the always_inline attribute is not a sufficient solution for code inlining problem.</p>
<p>Still, it is not clear why GCC estimates that kmalloc() would be compiled into many instructions. As shown, it is compiled into a single call to __kmalloc(). To answer this question, we need to follow kmalloc() code, which eventually uses the ilog2() macro to compute the log2 of an integer, in order to compute the page allocation order.</p>
<p>Here is a and shortened version of ilog2():</p>
<figure class="highlight"><pre><code class="language-c" data-lang="c"><span class="cp">#define ilog2(n) \
( \
__builtin_constant_p(n) ? ( \
</span><span class="cm">/* Optimized version for constants */</span><span class="cp"> \
(n) < 2 ? 0 : \
(n) & (1ULL << 63) ? 63 : \
(n) & (1ULL << 62) ? 62 : \
</span><span class="cm">/* ... */</span><span class="cp"> \
(n) & (1ULL << 3) ? 3 : \
(n) & (1ULL << 2) ? 2 : \
1 ) : \
</span><span class="cm">/* Another version for non-constants */</span><span class="cp"> \
(sizeof(n) <= 4) ? \
__ilog2_u32(n) : \
__ilog2_u64(n) \
}</span></code></pre></figure>
<p>As shown, the macro first uses the built-in function __builtin_constant_p() to determine whether n is known to be a constant during compilation time. If n is known to be constant, a long series of conditions is evaluated to compute the result during compilation time, which allows further optimizations. Otherwise, if n is not known to be constant, a short code is emitted to compute during runtime the result. Yet, regardless of whether n is constant or not, all of the conditions in the ilog2() macro are evaluated during compilation time and do not translate into any machine code instructions.</p>
<p>However, although the generated code is efficient, it causes GCC, again, to estimate the number of instructions that ilog2() takes incorrectly. Apparently, the number of instructions is estimated before inlining decisions take place, and in this stage the compiler usually still does not know whether n is constant. Later, after inlining decisions are performed, GCC cannot update the instruction count estimation accordingly.</p>
<p>This inlining problem is not as common as the previous one, yet it is not rare. Bit operations (e.g., test_bit()) and bitmaps commonly use __builtin_constant_p() in the described manner. As a result, functions that use these facilities, for example cpumask_weight(), are not inlined.</p>
<p>A possible solution for this problem is to use the built-in __builtin_choose_expr() to test __builtin_constant_p() instead of using C if-conditions and conditional operators (?:) :</p>
<figure class="highlight"><pre><code class="language-c" data-lang="c"><span class="cp">#define ilog2(n) \
( \
__builtin_choose_expr(__builtin_constant_p(n), \
((n) < 2 ? 0 : \
(n) & (1ULL << 63) ? 63 : \
(n) & (1ULL << 62) ? 62 : \
</span><span class="cm">/* ... */</span><span class="cp"> \
(n) & (1ULL << 3) ? 3 : \
(n) & (1ULL << 2) ? 2 : \
1 )), \
(sizeof(n) <= 4) ? \
__ilog2_u32(n) : \
__ilog2_u64(n) \
}</span></code></pre></figure>
<p>This built-in is evaluated earlier in the compilation process, before inlining decisions are being made. Yet, there is a catch: as this built-in is evaluated earlier, GCC is only able to determine that an argument is constant for constant expressions, which can cause less efficient code to be generated. For instance, if a constant was given as a function argument, GCC will not be able to determine it is constant. In the following case, for example, the non-constant version will be used:</p>
<figure class="highlight"><pre><code class="language-c" data-lang="c"><span class="kt">int</span> <span class="nf">bar</span><span class="p">(</span><span class="kt">int</span> <span class="n">n</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">ilog2</span><span class="p">(</span><span class="n">n</span><span class="p">)</span>
<span class="p">}</span>
<span class="kt">int</span> <span class="nf">foo</span><span class="p">(</span><span class="kt">int</span> <span class="n">n</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">bar</span><span class="p">(</span><span class="n">n</span><span class="p">);</span>
<span class="p">}</span>
<span class="n">v</span> <span class="o">=</span> <span class="n">foo</span><span class="p">(</span><span class="n">bar</span><span class="p">(</span><span class="mi">5</span><span class="p">));</span> <span class="cm">/* will use the non-constant version */</span></code></pre></figure>
<p>It is therefore questionable whether using __builtin_choose_expr() is an appropriate solution. Perhaps it is better to just mark functions such as kzalloc() with the always_inline attribute. Compiling using LLVM reveals, again, that LLVM inlining decisions are not negatively affected by the use of __builtin_constant_p().</p>
<h2 id="function-attributes">Function attributes</h2>
<p>Finally, there are certain function attributes that affect inlining decision. Using function attributes to set an optimization levels for specific functions can prevent the compiler from inlining the functions or functions that are called by them. The Linux kernel rarely uses such attributes, but one of its uses is in the KVM function <a href="https://elixir.bootlin.com/linux/v4.17/source/arch/x86/kvm/vmx.c#L9669">vmx_vcpu_run()</a> which is a very hot function that launches or resumes the virtual machine. The use of the optimization attribute in this function is actually just to prevent cloning of the function. Its side-effect is, however, that all the functions it uses are not inlined, including, for example the function <a href="https://elixir.bootlin.com/linux/v4.17/source/arch/x86/kvm/vmx.c#L706">to_vmx()</a>:</p>
<figure class="highlight"><pre><code class="language-objdump" data-lang="objdump">0x0000000000000150 <+0>: push %rbp
0x0000000000000151 <+1>: mov %rdi,%rax
0x0000000000000154 <+4>: mov %rsp,%rbp
0x0000000000000157 <+7>: pop %rbp
0x0000000000000158 <+8>: retq </code></pre></figure>
<p>This function just returns as an output the same argument it got as an input. Not inlining functions that are called by vmx_vcpu_run() induces significant overhead, which can be as high as 10% for a VM-exit.</p>
<p>Finally, the cold function attribute causes inlining to be done less aggressively. This attribute informs the compiler that a function is unlikely to be executed, and the compiler, among other things, optimizes these functions for size rather than speed, which can result in very non-aggressive inlining decisions. All the __init and __exit functions, which are used during the kernel and modules (de)initializations are marked as cold. It is questionable whether this is the desired behavior.</p>
<h2 id="conclusions">Conclusions</h2>
<p>Despite the fact that C appears to give us great control over the generated code, it is not always the case. Compiler extensions may be needed to give programmers greater control. Tools that analyze whether the generated binary is efficient, considering the source code, may be needed. In the meanwhile, there is no alternative to manual inspection of the generated binary code.</p>
<p>Thanks to Linus Torvalds, Hans Peter Anvin, Masahiro Yamada, Josh Poimboeuf, Peter Zijistra, Kees Cook, Ingo Molnar and others for their assistance in the analysis and in solving this problem.</p>]]></content><author><name></name></author><category term="Linux" /><summary type="html"><![CDATA[The Linux kernel strives to be fast and efficient. As it is written mostly in C, it can mostly control how the generated machine code looks. Nevertheless, as the kernel code is compiled into machine code, the compiler optimizes the generated code to improve its performance. The kernel code, however, employs uncommon coding techniques, which can fail code optimizations. In this blog-post, I would share my experience in analyzing the reasons for poor code inlining of the kernel code. Although the performance improvement are not significant in most cases, understanding these issues are valuable in preventing them from becoming larger. New-lines, as promised, will be one of the reasons, though not the only one.]]></summary></entry><entry><title type="html">Can you break Windows Page Table Isolation?</title><link href="https://nadav.amit.zone/windows/2018/09/16/windows-pti.html" rel="alternate" type="text/html" title="Can you break Windows Page Table Isolation?" /><published>2018-09-16T00:52:01+03:00</published><updated>2018-09-16T00:52:01+03:00</updated><id>https://nadav.amit.zone/windows/2018/09/16/windows-pti</id><content type="html" xml:base="https://nadav.amit.zone/windows/2018/09/16/windows-pti.html"><![CDATA[<p>The Meltdown and L1TF attacks proved that Intel CPUs are susceptible for attacks that use speculative execution to leak data from the kernel address-space. The root-cause for both of these attacks is the ability of unprivileged code to read speculatively data even if the page-table permissions should prevent the access. To address this security vulnerability Page Table Isolation (PTI) has been introduced by all common OSes. When PTI is used, a different set of page-tables is used when user-space run.</p>
<p>The key for PTI to be effective is that it only allows an access to minimal set of privileged pages, whose content can be leaked. These pages include memory that holds various privileged architectural data-structures, for example the Global Descriptor Table (GDT), and privileged trampoline code. This trampoline code runs when a system call, interrupt or exception are invoked, and switches to the kernel page-table hierarchy, for the kernel to be able to handle this event.</p>
<p>Is PTI implemented correctly? Well, in Linux, PTI implementation is open for review, reasonable, and is easy to audit . Microsoft provides a lot of information about the implementation of PTI in their system, but the code is obviously not open for review. Out of curiosity, I decided to check Windows 10. Using KVM and a simple script, I dumped pages which are mapped as supervisor pages in the userspace. The results were surprising:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">sudo</span> ./pti_test.py <span class="nt">--socket</span> ~/vm-images/qmp-sock
kernel frames: 124
page tables accessible by the user: 0
page tables accessible by the kernel: 115
writable page tables: 115
page tables: 115
global kernel frames: 9
kernel w+x: 118
kernel/user aliases: 0</code></pre></figure>
<p>This results surprised me for two reasons. First, Windows 10 does not implement <a href="https://en.wikipedia.org/wiki/W%5EX">W^X</a>, and maps pages which are both executable and writable even in the user page-tables. Inspecting the content of these pages shows that at least some of them are guaranteed not to hold code, and should have been marked as non-executable in the page-tables. I contacted Microsoft which claimed that this is intended since “in some cases the kernel is mapped with large pages” and that this can be prevented by enabling <a href="https://docs.microsoft.com/en-us/windows/security/threat-protection/windows-defender-exploit-guard/enable-virtualization-based-protection-of-code-integrity">virtualization based protection (VBS)</a>. However, dumping the page-tables (you can use the attached script), shows that no huge-pages were used, at least in my experiment, and still pages were mapped as writable and executable. VBS is not commonly used, and it might induce significant performance overheads, especially when Windows is being run inside a virtual-machine.</p>
<p>The second issue is that the page-tables themselves are mapped in the guest page-tables. Actually, Microsoft previously had a bug in which these page-tables could also be modified by userspace applications (in some Windows version). This obviously was a terrible <a href="https://www.bleepingcomputer.com/news/microsoft/meltdown-patch-opened-bigger-security-hole-on-windows-7/">security vulnerability</a>, since userspace applications could have established page-table entries that would allow them to access (read or write) any piece of memory. Today, they keep being mapped as writable, but only accessible by privileged code. Microsoft, again, claimed that this is necessary. I cannot understand why - Linux, for example, does not need to do so.</p>
<p>Can anyone exploit these behavior? Well, besides the fact that the lack of W^X ease fully compromising a system when another security vulnerability is found, Windows behavior might potentially enable more Meltdown-like attacks. As shown by the Meltdown and L1TF vulnerabilities, Intel CPUs defer at least some PTE validity and permission checks and still uses them during speculative execution. Mapping as few pages as possible in the userspace page-tables is necessary for good Meltdown mitigation as Microsoft itself <a href="https://blogs.technet.microsoft.com/srd/2018/03/23/kva-shadow-mitigating-meltdown-on-windows/">acknowledges</a>. If one can somehow speculatively inject code or modify the page tables or the executable code, and use the modified versions, this would cause a security vulnerability.</p>
<p>Still, it is not clear whether it actually poses a security vulnerability. Initially, I thought that mapping the page-tables might be exploited by running a speculatively-executed code that sets page-table entries that grant userspace code access to privileged data, and then reads the data and leaks it. However, this is unlikely to work. Accessing data can only be done after the page-table entry is cached in the translation lookaside buffer (TLB), and the TLB should not be able to hold entries whose content is based on speculative execution. If TLB entries would have been set based on speculative execution results, this would cause correctness issues.</p>
<p>Feel free to use my silly script (through this <a href="/downloads/pti_test.py">link</a>) and let me know what you think. And if you are a security researcher, it might be worthy to have a look at Windows Spectre v2 mitigations. As I noted before, performance counters indicate that there are indirect branches inside Windows kernel, which might not be safe if Windows relies solely on retpolines for Spectre v2.</p>]]></content><author><name></name></author><category term="Windows" /><summary type="html"><![CDATA[The Meltdown and L1TF attacks proved that Intel CPUs are susceptible for attacks that use speculative execution to leak data from the kernel address-space. The root-cause for both of these attacks is the ability of unprivileged code to read speculatively data even if the page-table permissions should prevent the access. To address this security vulnerability Page Table Isolation (PTI) has been introduced by all common OSes. When PTI is used, a different set of page-tables is used when user-space run.]]></summary></entry><entry><title type="html">roundup() vs round_up()</title><link href="https://nadav.amit.zone/linux/2018/09/02/roundup.html" rel="alternate" type="text/html" title="roundup() vs round_up()" /><published>2018-09-02T00:52:01+03:00</published><updated>2018-09-02T00:52:01+03:00</updated><id>https://nadav.amit.zone/linux/2018/09/02/roundup</id><content type="html" xml:base="https://nadav.amit.zone/linux/2018/09/02/roundup.html"><![CDATA[<p>Occasionally, you make a mistake that makes you think that surely others have made a similar one.</p>
<p>Such case happened to me one day when I was trying, in Linux, to round up a number (x) into a multiple of another number (y). The Linux kernel provides two macros for such an operation:</p>
<ul>
<li>
<p><a href="https://elixir.bootlin.com/linux/v4.17/source/include/linux/kernel.h#L87">round_up(x, y)</a> which only works when y is a power of 2, as it does the rounding using bit operations for efficiency. If y is not a power of 2, it quietly returns the wrong result.</p>
</li>
<li>
<p><a href="https://elixir.bootlin.com/linux/v4.17/source/include/linux/kernel.h#L113">roundup(x, y)</a> which works for any value of y.</p>
</li>
</ul>
<p>Yes, this is not a typo. One macro is named round_up() and the other roundup(). Both macros are defined in the same file (kernel.h), so their developers should have known that there are similarly named macros. They are not defined, however, next to each other and there is no comment in the code to explain the difference between the two, so it is very easy for a developer to use the wrong macro. There is also no assertion to ensure that y is a power of 2 in round_up(), and unfortunately, adding such an assertion is not as simple as adding a single line, since the macro is also used to define the size of some arrays.</p>
<p>Both macros are widely used (~500 times each), which made me suspect that in some cases round_up() might have been used inappropriately (i.e., when y is not a power of 2). Out of curiosity, I added a static assertion just for the cases in which y is known to be constant in compilation time. Immediately, two bugs were found, one in a <a href="https://lore.kernel.org/patchwork/patch/913171/">GPIO driver</a> and a second one in the <a href="https://patchwork.kernel.org/patch/10447343/">USB video class driver</a>. The impact of these bugs is not clear. There might actually be more bugs in cases where y is not known during compilation time or when a different kernel configuration is used.</p>
<p>A less lazy person would have written a <a href="http://coccinelle.lip6.fr/">Coccinelle</a> script to rename one of the functions and to add the required assertions, but making such tree-wide changes is a time consuming task. I have just submitted patches to fix the issues I found.</p>
<p>People can take different lessons from this story. Personally, it just made me think: how many more silly bugs exist that can easily be prevented by adding simple assertions?</p>]]></content><author><name></name></author><category term="Linux" /><summary type="html"><![CDATA[Occasionally, you make a mistake that makes you think that surely others have made a similar one.]]></summary></entry><entry><title type="html">Are you protected against Spectre &amp; Meltdown? Probably not</title><link href="https://nadav.amit.zone/linux/2018/08/02/spectrev2-protection.html" rel="alternate" type="text/html" title="Are you protected against Spectre &amp; Meltdown? Probably not" /><published>2018-08-02T00:52:01+03:00</published><updated>2018-08-02T00:52:01+03:00</updated><id>https://nadav.amit.zone/linux/2018/08/02/spectrev2-protection</id><content type="html" xml:base="https://nadav.amit.zone/linux/2018/08/02/spectrev2-protection.html"><![CDATA[<p>Several months ago, everybody was surprised to find out that the CPU speculative execution can be exploited to leak privileged information, using attacks that were named Spectre and Meltdown. Technical blogs and even the main-stream media reported broadly about those vulnerabilities. CPU vendors have struggled to provide firmware patches that would prevent the attacks in a timely manner. OS and other software providers introduced software solutions such as the <a href="https://software.intel.com/security-software-guidance/api-app/sites/default/files/Retpoline-A-Branch-Target-Injection-Mitigation.pdf">retpoline</a> and page table isolation (PTI) to protect against these vulnerabilities. All of these mitigations caused performance degradation, required extensive engineering effort, and caused various <a href="https://www.darkreading.com/risk/microsoft-issues-emergency-patch-to-disable-intels-broken-spectre-fix/d/d-id/1330932">problems</a>.</p>
<p>So half a year later - are you protected? Probably not. Recently I found that the Linux protection against Spectre v2 is broken in virtual machines and Dave Hansen found a bug in the <a href="https://www.darkreading.com/risk/microsoft-issues-emergency-patch-to-disable-intels-broken-spectre-fix/d/d-id/1330932">Meltdown</a> protection. Spectre v1 was never considered fully resolved, with mitigations keep coming in, but even the existing ones were found to be <a href="https://www.spinics.net/lists/linux-tip-commits/msg44433.html">buggy</a>.</p>
<p>There is clearly a problem, as currently it is hard for people to easily realize whether they are protected against these attacks. Sure, one can see whether the OS or other piece of software reports that it is protected, but these reports might be wrong. There is a fundamental problem with these protections that, in a way, is the same one that caused Linus Torvalds to politely (yes, politely) <a href="https://lkml.org/lkml/2018/2/15/1378">decline</a> a Meltdown mitigation technique we proposed<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>:</p>
<blockquote>
<p>Sure, I can see it working, but it’s some really shady stuff, and now the scheduler needs to save/restore/check one more subtle bit.</p>
<p>And if you get it wrong, things will happily work, except you’ve now defeated PTI. But you’ll never notice, because you won’t be testing for it, and the only people who will are the black hats.</p>
<p>This is exactly the “security depends on it being in sync” thing that makes me go “eww” about the whole model. Get one thing wrong, and you’ll blow all the PTI code out of the water.</p>
</blockquote>
<p>Linus criticism of our work is valid, yet it does not seem that other protection mechanisms against these vulnerabilities are much better. And even if the OS is well-protected against these vulnerabilities, nobody guarantees that the system will remain safe after an out-of-tree module is loaded, for example. All it takes for the Spectre v2 protection to be broken, for example, is a single indirect branch that was not converted into a retpoline.</p>
<p>It seems that in order to make the protection work, independent tools that validate the protection mechanisms are needed. I found the Spectre v2 by using the hardware performance counters to count indirect branches that were executed by the kernel and finding it is not zero. Dumping the page-tables and tracing translation-lookaside buffer (TLB) invalidations can be used to find out PTI bugs. Anti-malware tools should take up the glove and make these checks.</p>
<p>Yet, perhaps there is an additional problem of over-hyping Spectre. Side-channel attacks were known long before Spectre, and invoking them using speculative execution may not be such a game-changer. Unlike Meltdown, which is a real CPU bug, the Spectre family of vulnerabilities may pose lower risk as they are not easily exploitable. The Spectre v2 proof-of-concept exploited some Linux wrongdoings (e.g., not zeroing registers after a context switch from a virtual machine), which were relatively easily fixed and became a good mitigation against other OS bugs. Some new Spectre attacks were not even reported to be successfully exploitable other than in artificial <a href="https://bugs.chromium.org/p/project-zero/issues/detail?id=1528">demos</a>.</p>
<p>It might be that the industry over-reacted to Spectre. Even if Spectre vulnerabilities are addressed, software might still leak privileged data through side-channels, so it is not as if the existing protection schemes are complete. Now that the media frenzy is gone, perhaps it is time to reconsider whether paying in performance for questionable “generic” protection schemes against these attacks makes sense, or whether protection should be done on a case-by-case basis.</p>
<p>Update: I wonder if Windows is indeed safe. Windows uses retpoline, but it is not clear whether they are used exclusively or with alternative solutions that use hardware mitigation (IBPB/IBRS). Anyhow, measuring the performance counter of Windows 10 (that runs in the VM) raises some questions, as it show there are indirect branches inside Windows kernel. Here are the performance counters as measure in a KVM guest:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">sudo </span>perf <span class="nb">stat</span> <span class="nt">-e</span> br_inst_exec.taken_indirect_jump_non_call_ret:Gk <span class="se">\</span>
<span class="nt">-e</span> br_inst_exec.taken_indirect_near_call:Gk <span class="nt">-a</span> <span class="nt">--</span> <span class="nb">sleep </span>5
Performance counter stats <span class="k">for</span> <span class="s1">'system wide'</span>:
1,682,939 br_inst_exec.taken_indirect_jump_non_call_ret:Gk
1,102,037 br_inst_exec.taken_indirect_near_call:Gk
5.001077704 seconds <span class="nb">time </span>elapsed</code></pre></figure>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>Based on work with Michael Wei and Dan Tsafrir (who may not share my views) <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>]]></content><author><name></name></author><category term="Linux" /><summary type="html"><![CDATA[Several months ago, everybody was surprised to find out that the CPU speculative execution can be exploited to leak privileged information, using attacks that were named Spectre and Meltdown. Technical blogs and even the main-stream media reported broadly about those vulnerabilities. CPU vendors have struggled to provide firmware patches that would prevent the attacks in a timely manner. OS and other software providers introduced software solutions such as the retpoline and page table isolation (PTI) to protect against these vulnerabilities. All of these mitigations caused performance degradation, required extensive engineering effort, and caused various problems.]]></summary></entry></feed>