https://www.chiliproject.org/2011-02-10T08:01:09+01:00ChiliProjectChiliProject - Feature #169: [PATCH] hiding form pages from search engines https://www.chiliproject.org/issues/169?journal_id=98702011-02-10T08:01:09+01:00Yuki Sonodayugui@yugui.jp
<ul><li><strong>Start date</strong> set to <i>2011-02-10</i></li><li><strong>Estimated time</strong> deleted ()</li><li><strong>Priority</strong> changed from <i>Normal</i> to <i>Normal</i></li><li><strong>Project</strong> set to <i>ChiliProject</i></li><li><strong>Target version</strong> set to <i>1.1.0 — Bell</i></li><li><strong>Assignee</strong> set to <i>Eric Davis</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>0</i></li><li><strong>Subject</strong> set to <i>[PATCH] hiding form pages from search engines </i></li><li><strong>Category</strong> set to <i>User interface</i></li><li><strong>Tracker</strong> set to <i>Feature</i></li><li><strong>Due date</strong> deleted ()</li><li><strong>Subproject of</strong> deleted ()</li><li><strong>Description</strong> set to <i>Form pages like /issues/new are not worth to be indexed by search engines. An...</i> <a href="/journals/9870/diff/description" class="lightbox-ajax">More</a></li><li><strong>Status</strong> changed from <i>Open</i> to <i>Closed</i></li></ul> ChiliProject - Feature #169: [PATCH] hiding form pages from search engines https://www.chiliproject.org/issues/169?journal_id=9462011-02-10T09:52:16+01:00Felix Schäferfelix+chili@oh14.de
<ul></ul><p>I guess having something like <code>*/new</code> in the robots.txt wouldn't work, would it?</p> ChiliProject - Feature #169: [PATCH] hiding form pages from search engines https://www.chiliproject.org/issues/169?journal_id=9532011-02-10T13:07:05+01:00Yuki Sonodayugui@yugui.jp
<ul></ul><p>According to <a class="external" href="http://www.robotstxt.org/robotstxt.html">http://www.robotstxt.org/robotstxt.html</a>, robots.txt does not support glob. So we can not expect */new works fine.</p> ChiliProject - Feature #169: [PATCH] hiding form pages from search engines https://www.chiliproject.org/issues/169?journal_id=9622011-02-10T23:36:24+01:00Eric Davis
<ul></ul><p>I think this is a good idea. I'd like to improve on it a little bit though by making the robot_exclusion_tag take options for the content section (e.g. <code>robot_exclusion_tag("NOINDEX,FOLLOW,NOARCHIVE")</code> or <code>robot_exclusion_tag("NOINDEX,NOFOLLOW")</code>). Then we (or plugins) could have more control over the indexing options for each page.</p>
<p>Thoughts?</p> ChiliProject - Feature #169: [PATCH] hiding form pages from search engines https://www.chiliproject.org/issues/169?journal_id=9722011-02-11T07:30:59+01:00Felix Schäferfelix+chili@oh14.de
<ul></ul><p>Eric Davis wrote:</p>
<blockquote>
<p>Thoughts?</p>
</blockquote>
<p>What about making <code>NOINDEX,FOLLOW,NOARCHIVE</code> the default and calling the method with any collection of <code>(NO)SOMETHING</code> overrides the default for that keyword?</p> ChiliProject - Feature #169: [PATCH] hiding form pages from search engines https://www.chiliproject.org/issues/169?journal_id=9872011-02-11T19:01:34+01:00Eric Davis
<ul></ul><p>This was my idea. It let us have more control of what the actual content is in case the meta tag allows other values later.</p>
<pre><code class="ruby syntaxhl"><span class="CodeRay"><span class="line-numbers">1</span> <span class="keyword">def</span> <span class="function">robot_exclusion_tag</span>(content=<span class="string"><span class="delimiter">"</span><span class="content">NOINDEX,FOLLOW,NOARCHIVE</span><span class="delimiter">"</span></span>)
<span class="line-numbers">2</span> <span class="string"><span class="delimiter">"</span><span class="content"><meta name='ROBOTS' content=</span><span class="inline"><span class="inline-delimiter">#{</span>content<span class="inline-delimiter">}</span></span><span class="content"> /></span><span class="delimiter">"</span></span>
<span class="line-numbers">3</span> <span class="keyword">end</span>
</span></code></pre> ChiliProject - Feature #169: [PATCH] hiding form pages from search engines https://www.chiliproject.org/issues/169?journal_id=9912011-02-11T21:31:41+01:00Felix Schäferfelix+chili@oh14.de
<ul></ul><p>Eric Davis wrote:</p>
<blockquote>
<p>This was my idea. It let us have more control of what the actual content is in case the meta tag allows other values later.</p>
</blockquote>
<p>No, I meant having <code>NOINDEX,FOLLOW,NOARCHIVE</code> be the default, and if you call it with <code>INDEX</code> to get <code>INDEX,FOLLOW,NOARCHIVE</code>. I just realized that's overengineering it though, I like your proposal :-)</p> ChiliProject - Feature #169: [PATCH] hiding form pages from search engines https://www.chiliproject.org/issues/169?journal_id=9972011-02-11T23:54:22+01:00Eric Davis
<ul><li><strong>Target version</strong> set to <i>1.1.0 — Bell</i></li><li><strong>Assignee</strong> set to <i>Eric Davis</i></li></ul><p>Yea I thought about doing keywords too but then we would have to maintain a list of valid ones. Hence the idea of just using a simple string.</p>
<p>I'll add and modify this patch. I think it's minor enough for 1.1.0.</p> ChiliProject - Feature #169: [PATCH] hiding form pages from search engines https://www.chiliproject.org/issues/169?journal_id=10202011-02-14T02:20:35+01:00Eric Davis
<ul><li><strong>Status</strong> changed from <i>Open</i> to <i>Ready for review</i></li></ul><p>I've modified Yuki Sonoda's patch and the code is ready for review.</p>
<p><a class="external" href="https://github.com/chiliproject/chiliproject/pull/7">https://github.com/chiliproject/chiliproject/pull/7</a></p> ChiliProject - Feature #169: [PATCH] hiding form pages from search engines https://www.chiliproject.org/issues/169?journal_id=10222011-02-14T07:05:40+01:00Felix Schäferfelix+chili@oh14.de
<ul></ul><p>Looks good to me. I'll merge it by the time I'm around a more stable connection if you haven't done so until then.</p> ChiliProject - Feature #169: [PATCH] hiding form pages from search engines https://www.chiliproject.org/issues/169?journal_id=10322011-02-14T09:33:38+01:00Holger Justchiliproject@meine-er.de
<ul></ul><p>I still like Felix' idea of having defaults and being able to gradually overwrite them. This could be done like this:</p>
<pre><code class="ruby syntaxhl"><span class="CodeRay"><span class="line-numbers"> 1</span><span class="comment"># Add a HTML meta tag to control robots (web spiders)</span>
<span class="line-numbers"> 2</span><span class="comment">#</span>
<span class="line-numbers"> 3</span><span class="comment"># @param [optional, String] changed content of the ROBOTS tag.</span>
<span class="line-numbers"> 4</span><span class="comment"># defaults to no index, follow, and no archive</span>
<span class="line-numbers"> 5</span><span class="keyword">def</span> <span class="function">robot_exclusion_tag</span>(content=<span class="string"><span class="delimiter">"</span><span class="delimiter">"</span></span>)
<span class="line-numbers"> 6</span> default_content = { <span class="string"><span class="delimiter">"</span><span class="content">INDEX</span><span class="delimiter">"</span></span> => <span class="string"><span class="delimiter">"</span><span class="content">NO</span><span class="delimiter">"</span></span>,
<span class="line-numbers"> 7</span> <span class="string"><span class="delimiter">"</span><span class="content">FOLLOW</span><span class="delimiter">"</span></span> => <span class="string"><span class="delimiter">"</span><span class="delimiter">"</span></span>,
<span class="line-numbers"> 8</span> <span class="string"><span class="delimiter">"</span><span class="content">ARCHIVE</span><span class="delimiter">"</span></span> => <span class="string"><span class="delimiter">"</span><span class="content">NO</span><span class="delimiter">"</span></span> }
<span class="line-numbers"> 9</span>
<span class="line-numbers"><strong>10</strong></span> args = content.upcase.split(<span class="string"><span class="delimiter">"</span><span class="content">,</span><span class="delimiter">"</span></span>).inject({}) <span class="keyword">do</span> |args, arg|
<span class="line-numbers">11</span> value = arg.gsub(<span class="regexp"><span class="delimiter">/</span><span class="content">^(NO)</span><span class="delimiter">/</span></span>, <span class="string"><span class="delimiter">"</span><span class="delimiter">"</span></span>)
<span class="line-numbers">12</span> args[value] = <span class="global-variable">$1</span> || <span class="string"><span class="delimiter">"</span><span class="delimiter">"</span></span>
<span class="line-numbers">13</span> args
<span class="line-numbers">14</span> <span class="keyword">end</span>
<span class="line-numbers">15</span> default_content.merge(args).collect{ |k, v| v+k }.join(<span class="string"><span class="delimiter">"</span><span class="content">,</span><span class="delimiter">"</span></span>)
<span class="line-numbers">16</span><span class="keyword">end</span>
</span></code></pre> ChiliProject - Feature #169: [PATCH] hiding form pages from search engines https://www.chiliproject.org/issues/169?journal_id=10402011-02-14T23:07:28+01:00Eric Davis
<ul></ul><p>Holger Just wrote:</p>
<blockquote>
<p>I still like Felix' idea of having defaults and being able to gradually overwrite them.</p>
</blockquote>
<p>Just seems like a lot of code to me that might not be used that often.</p> ChiliProject - Feature #169: [PATCH] hiding form pages from search engines https://www.chiliproject.org/issues/169?journal_id=10502011-02-15T06:28:50+01:00Felix Schäferfelix+chili@oh14.de
<ul></ul><p>Eric Davis wrote:</p>
<blockquote>
<p>Just seems like a lot of code to me that might not be used that often.</p>
</blockquote>
<p>That's what I meant with "don't overengineer it" ;-) I think the simple version is fine, avoiding to have to write it all out for those few times you need other params is not worth it.</p> ChiliProject - Feature #169: [PATCH] hiding form pages from search engines https://www.chiliproject.org/issues/169?journal_id=10802011-02-17T01:12:06+01:00Eric Davis
<ul><li><strong>Status</strong> changed from <i>Ready for review</i> to <i>Closed</i></li></ul><p>Merged into master for 1.1.0. Thank you for the patch Yuki Sonoda.</p>