{"id":21553,"date":"2024-07-10T11:18:20","date_gmt":"2024-07-10T11:18:20","guid":{"rendered":"https:\/\/marketist.co\/blog\/?p=21553"},"modified":"2025-08-30T04:19:52","modified_gmt":"2025-08-30T04:19:52","slug":"how-to-add-a-robots-txt-file-to-a-custom-website-in-php","status":"publish","type":"post","link":"https:\/\/marketist.co\/blog\/how-to-add-a-robots-txt-file-to-a-custom-website-in-php\/","title":{"rendered":"How to Add a robots.txt File to a Custom Website in PHP"},"content":{"rendered":"<p>Adding a <code>robots.txt<\/code> file to a static website is a straightforward process. This file helps guide search engine crawlers on which parts of your site should be indexed. Here\u2019s how you can create and add one to your PHP-based static website.<\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_80 ez-toc-wrap-center counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 eztoc-toggle-hide-by-default' ><ul class='ez-toc-list-level-4' ><li class='ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/marketist.co\/blog\/how-to-add-a-robots-txt-file-to-a-custom-website-in-php\/#Step_1_Create_the_robotstxt_File\" >Step 1: Create the robots.txt File<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/marketist.co\/blog\/how-to-add-a-robots-txt-file-to-a-custom-website-in-php\/#Step_2_Define_the_Rules_for_Crawlers\" >Step 2: Define the Rules for Crawlers<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/marketist.co\/blog\/how-to-add-a-robots-txt-file-to-a-custom-website-in-php\/#Step_3_Upload_the_robotstxt_File\" >Step 3: Upload the robots.txt File<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/marketist.co\/blog\/how-to-add-a-robots-txt-file-to-a-custom-website-in-php\/#Step_4_Verify_the_robotstxt_File\" >Step 4: Verify the robots.txt File<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/marketist.co\/blog\/how-to-add-a-robots-txt-file-to-a-custom-website-in-php\/#Step_5_Test_with_Google_Search_Console\" >Step 5: Test with Google Search Console<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/marketist.co\/blog\/how-to-add-a-robots-txt-file-to-a-custom-website-in-php\/#Conclusion\" >Conclusion<\/a><\/li><\/ul><\/nav><\/div>\n<h4><span class=\"ez-toc-section\" id=\"Step_1_Create_the_robotstxt_File\"><\/span>Step 1: Create the <code>robots.txt<\/code> File<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>Create a new file named <code>robots.txt<\/code> in the root directory of your website.<\/p>\n<h4><span class=\"ez-toc-section\" id=\"Step_2_Define_the_Rules_for_Crawlers\"><\/span>Step 2: Define the Rules for Crawlers<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>Open <code>robots.txt<\/code> in a text editor and add the rules. Here is an example of a basic <code>robots.txt<\/code> file:<\/p>\n<p><img decoding=\"async\" class=\"alignnone size-medium wp-image-21554\" src=\"https:\/\/marketist.co\/blog\/wp-content\/uploads\/2024\/07\/robotstxt-300x74.png\" alt=\"\" width=\"300\" height=\"74\" srcset=\"https:\/\/marketist.co\/blog\/wp-content\/uploads\/2024\/07\/robotstxt-300x74.png 300w, https:\/\/marketist.co\/blog\/wp-content\/uploads\/2024\/07\/robotstxt.png 526w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/><\/p>\n<h4><span class=\"ez-toc-section\" id=\"Step_3_Upload_the_robotstxt_File\"><\/span>Step 3: Upload the <code>robots.txt<\/code> File<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>Upload the <code>robots.txt<\/code> file to the root directory of your website using an FTP client or your web hosting control panel.<\/p>\n<h4><span class=\"ez-toc-section\" id=\"Step_4_Verify_the_robotstxt_File\"><\/span>Step 4: Verify the <code>robots.txt<\/code> File<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>Visit <code>https:\/\/yourwebsite.com\/robots.txt<\/code> in your browser to ensure it\u2019s accessible and displays the correct rules.<\/p>\n<h4><span class=\"ez-toc-section\" id=\"Step_5_Test_with_Google_Search_Console\"><\/span>Step 5: Test with Google Search Console<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<ol>\n<li>Go to Google Search Console.<\/li>\n<li>Select your website.<\/li>\n<li>Navigate to <strong>Crawl<\/strong> &gt; <strong>robots.txt Tester<\/strong>.<\/li>\n<li>Ensure your <code>robots.txt<\/code> file is correct and functional.<\/li>\n<\/ol>\n<h3><span class=\"ez-toc-section\" id=\"Conclusion\"><\/span>Conclusion<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Adding a <code>robots.txt<\/code> file to your static PHP website helps control which parts of your site are crawled by search engines, improving your SEO and protecting sensitive areas of your site. Follow these steps to create and verify your <code>robots.txt<\/code> file, ensuring it&#8217;s correctly guiding search engine crawlers.<\/p>\n<hr \/>\n<p style=\"text-align: center;\">Subscribe to Our <a href=\"https:\/\/marketist.co\/blog\/\"><strong>Blog<\/strong><\/a> with your email (<strong>Is it too much too ask?<\/strong>);<br \/>\n<div class=\"mf-form-shortcode\">\r\n\t\t<div\r\n\t\t\tid=\"metform-wrap-21440-21440\"\r\n\t\t\tclass=\"mf-form-wrapper\"\r\n\t\t\tdata-form-id=\"21440\"\r\n\t\t\tdata-action=\"https:\/\/marketist.co\/blog\/wp-json\/metform\/v1\/entries\/insert\/21440\"\r\n\t\t\tdata-wp-nonce=\"f36e4e7978\"\r\n\t\t\tdata-form-nonce=\"3408353017\"\r\n\t\t\tdata-quiz-summery = \"false\"\r\n\t\t\tdata-save-progress = \"false\"\r\n\t\t\tdata-form-type=\"general-form\"\r\n\t\t\tdata-stop-vertical-effect=\"\"\r\n\t\t\t><\/div>\r\n\r\n\r\n\t\t<!----------------------------- \r\n\t\t\t* controls_data : find the the props passed indie of data attribute\r\n\t\t\t* props.SubmitResponseMarkup : contains the markup of error or success message\r\n\t\t\t* https:\/\/developer.mozilla.org\/en-US\/docs\/Web\/JavaScript\/Reference\/Template_literals\r\n\t\t--------------------------- -->\r\n\r\n\t\t\t\t<script type=\"text\/mf\" class=\"mf-template\">\r\n\t\t\tfunction controls_data (value){\r\n\t\t\t\tlet currentWrapper = \"mf-response-props-id-21440\";\r\n\t\t\t\tlet currentEl = document.getElementById(currentWrapper);\r\n\t\t\t\t\r\n\t\t\t\treturn currentEl ? currentEl.dataset[value] : false\r\n\t\t\t}\r\n\r\n\r\n\t\t\tlet is_edit_mode = '' ? true : false;\r\n\t\t\tlet message_position = controls_data('messageposition') || 'top';\r\n\r\n\t\t\t\r\n\t\t\tlet message_successIcon = controls_data('successicon') || '';\r\n\t\t\tlet message_errorIcon = controls_data('erroricon') || '';\r\n\t\t\tlet message_editSwitch = controls_data('editswitchopen') === 'yes' ? true : false;\r\n\t\t\tlet message_proClass = controls_data('editswitchopen') === 'yes' ? 'mf_pro_activated' : '';\r\n\t\t\t\r\n\t\t\tlet is_dummy_markup = is_edit_mode && message_editSwitch ? true : false;\r\n\r\n\t\t\t\r\n\t\t\treturn html`\r\n\t\t\t\t<form\r\n\t\t\t\t\tclassName=\"metform-form-content\"\r\n\t\t\t\t\tref=${parent.formContainerRef}\r\n\t\t\t\t\tonSubmit=${ validation.handleSubmit( parent.handleFormSubmit ) }\r\n\t\t\t\t\r\n\t\t\t\t\t>\r\n\t\t\t\r\n\t\t\t\r\n\t\t\t\t\t${is_dummy_markup ? message_position === 'top' ?  props.ResponseDummyMarkup(message_successIcon, message_proClass) : '' : ''}\r\n\t\t\t\t\t${is_dummy_markup ? ' ' :  message_position === 'top' ? props.SubmitResponseMarkup`${parent}${state}${message_successIcon}${message_errorIcon}${message_proClass}` : ''}\r\n\r\n\t\t\t\t\t<!--------------------------------------------------------\r\n\t\t\t\t\t*** IMPORTANT \/ DANGEROUS ***\r\n\t\t\t\t\t${html``} must be used as in immediate child of \"metform-form-main-wrapper\"\r\n\t\t\t\t\tclass otherwise multistep form will not run at all\r\n\t\t\t\t\t---------------------------------------------------------->\r\n\r\n\t\t\t\t\t<div className=\"metform-form-main-wrapper\" key=${'hide-form-after-submit'} ref=${parent.formRef}>\r\n\t\t\t\t\t${html`\r\n\t\t\t\t\t\t<style key=\"1\">.elementor-21440 .elementor-element.elementor-element-8894fce.elementor-column > .elementor-widget-wrap{justify-content:center;}.elementor-21440 .elementor-element.elementor-element-5a26283{width:var( --container-widget-width, 50% );max-width:50%;--container-widget-width:50%;--container-widget-flex-grow:0;}.elementor-21440 .elementor-element.elementor-element-5a26283 > .elementor-widget-container{margin:0px 0px 0px 0px;padding:0px 0px 0px 0px;border-radius:20px 20px 20px 20px;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input, .elementor-21440 .elementor-element.elementor-element-5a26283 .elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input-wrapper .iti--separate-dial-code .iti__selected-flag, .elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input-wrapper .iti--separate-dial-code .iti__selected-dial-code{color:#000000;}.elementor-21440 .elementor-element.elementor-element-5a26283 .irs--round .irs-bar, .elementor-21440 .elementor-element.elementor-element-5a26283 .irs--round .irs-from, .elementor-21440 .elementor-element.elementor-element-5a26283 .irs--round .irs-to, .elementor-21440 .elementor-element.elementor-element-5a26283 .irs--round .irs-single{background-color:#000000;}.elementor-21440 .elementor-element.elementor-element-5a26283 .irs--round .irs-handle{border-color:#000000;}.elementor-21440 .elementor-element.elementor-element-5a26283 .irs--round .irs-from:before, .elementor-21440 .elementor-element.elementor-element-5a26283 .irs--round .irs-to:before, .elementor-21440 .elementor-element.elementor-element-5a26283 .irs--round .irs-single:before{border-top-color:#000000;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf_select__single-value{color:#000000;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input-wrapper{border-color:#000000;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input-wrapper .input-range__track--active{background-color:#000000;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input-wrapper .asRange .asRange-pointer .asRange-tip{background-color:#000000;border-color:#000000;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input-file-upload-label, .elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input-calculation-total{color:#000000;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input-file-upload-label svg path{stroke:#000000;fill:#000000;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf_multiselect__multi-value__label, .elementor-21440 .elementor-element.elementor-element-5a26283 .mf_multiselect__multi-value__remove,.elementor-21440 .elementor-element.elementor-element-5a26283 .mf_multiselect__menu-notice--no-options{color:#000000;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input:hover, .elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input-wrapper:hover .iti--separate-dial-code .iti__selected-flag, .elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input-wrapper:hover .iti--separate-dial-code .iti__selected-dial-code{color:#000000;}.elementor-21440 .elementor-element.elementor-element-5a26283 .irs--round .irs-handle:hover{border-color:#000000;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input-file-upload-label:hover{color:#000000;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input:hover .mf_select__single-value{color:#000000;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf-file-upload-container:hover .mf-input-file-upload-label svg path{stroke:#000000;fill:#000000;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf_multiselect__multi-value:hover .mf_multiselect__multi-value__label, .elementor-21440 .elementor-element.elementor-element-5a26283 .mf_multiselect__multi-value:hover .mf_multiselect__multi-value__remove,.elementor-21440 .elementor-element.elementor-element-5a26283 .mf_multiselect__menu-notice--no-options:hover{color:#000000;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input:focus, .elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input-wrapper:focus .iti--separate-dial-code .iti__selected-flag, .elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input-wrapper:focus .iti--separate-dial-code .iti__selected-dial-code{color:#000000;}.elementor-21440 .elementor-element.elementor-element-5a26283 .irs--round .irs-handle:focus{border-color:#000000;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf-file-upload-container:focus .mf-input-file-upload-label, .elementor-21440 .elementor-element.elementor-element-5a26283 .mf-file-upload-container:focus .mf-image-label, .elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input-calculation-total:focus{color:#000000;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input > .mf_select__control--is-focused .mf_select__single-value{color:#000000;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf-file-upload-container:focus .mf-input-file-upload-label svg path{stroke:#000000;fill:#000000;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input-calculation-total:focus, .elementor-21440 .elementor-element.elementor-element-5a26283 .mf_multiselect__option:focus, .elementor-21440 .elementor-element.elementor-element-5a26283 .mf_multiselect__multi-value:focus .mf_multiselect__multi-value__label, .elementor-21440 .elementor-element.elementor-element-5a26283 .mf_multiselect__multi-value:focus .mf_multiselect__multi-value__remove, .elementor-21440 .elementor-element.elementor-element-5a26283 .mf_multiselect__menu-notice--no-options:focus{color:#000000;}.elementor-21440 .elementor-element.elementor-element-5a26283 .input-range .input-range__track{border-radius:0px;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input{border-radius:0px;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf-file-upload-container .mf-input-file-upload-label, .elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input-wrapper .iti--separate-dial-code .iti__selected-flag, .elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input-calculation-total{border-radius:0px;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input-select > .mf_select__control, .elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input-multiselect .mf_multiselect__control, .elementor-21440 .elementor-element.elementor-element-5a26283 .mf_multiselect__option, .elementor-21440 .elementor-element.elementor-element-5a26283 .mf_multiselect__multi-value__label, .elementor-21440 .elementor-element.elementor-element-5a26283 .mf_multiselect__multi-value__remove, .elementor-21440 .elementor-element.elementor-element-5a26283 .mf_multiselect__menu-notice--no-options {border-radius:0px;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input:not([type=\"submit\"]):not([type=\"checkbox\"]):not([type=\"radio\"])::-webkit-input-placeholder{color:#c9c1c1;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input:not([type=\"submit\"]):not([type=\"checkbox\"]):not([type=\"radio\"])::-moz-placeholder{color:#c9c1c1;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input:not([type=\"submit\"]):not([type=\"checkbox\"]):not([type=\"radio\"]):-ms-input-placeholder{color:#c9c1c1;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input:not([type=\"submit\"]):not([type=\"checkbox\"]):not([type=\"radio\"]):-moz-placeholder{color:#c9c1c1;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf-time-input::before{color:#c9c1c1;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input::-webkit-input-placeholder{color:#c9c1c1;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input::-moz-placeholder{color:#c9c1c1;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input:-ms-input-placeholder{color:#c9c1c1;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf-input:-moz-placeholder{color:#c9c1c1;}.elementor-21440 .elementor-element.elementor-element-5a26283 :is(.mf_select__placeholder, .mf_multiselect__placeholder){color:#c9c1c1;}.elementor-21440 .elementor-element.elementor-element-5a26283 .mf-error-message{color:#FF0000;}.elementor-21440 .elementor-element.elementor-element-a015ee2 .mf-btn-wraper{text-align:center;}.elementor-21440 .elementor-element.elementor-element-a015ee2{width:auto;max-width:auto;}.elementor-21440 .elementor-element.elementor-element-a015ee2 > .elementor-widget-container{margin:0px 0px 0px 25px;}.elementor-21440 .elementor-element.elementor-element-a015ee2 .metform-btn{padding:15px 20px 15px 20px;color:#ffffff;border-style:none;border-radius:5px 5px 5px 5px;}.elementor-21440 .elementor-element.elementor-element-a015ee2 .metform-btn:hover{color:#FFFFFF;}.elementor-21440 .elementor-element.elementor-element-a015ee2 .metform-btn:focus{color:#FFFFFF;}<\/style>\t\t<div data-elementor-type=\"wp-post\" key=\"2\" data-elementor-id=\"21440\" className=\"elementor elementor-21440\">\n\t\t\t\t\t\t<section className=\"elementor-section elementor-top-section elementor-element elementor-element-3655339 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"3655339\" data-element_type=\"section\">\n\t\t\t\t\t\t<div className=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div className=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-8894fce\" data-id=\"8894fce\" data-element_type=\"column\">\n\t\t\t<div className=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div className=\"elementor-element elementor-element-5a26283 elementor-widget__width-initial elementor-widget elementor-widget-mf-email\" data-id=\"5a26283\" data-element_type=\"widget\" data-settings=\"{&quot;mf_input_name&quot;:&quot;mf-email&quot;}\" data-widget_type=\"mf-email.default\">\n\t\t\t\t<div className=\"elementor-widget-container\">\n\t\t\t\t\t\r\n\t\t<div className=\"mf-input-wrapper\">\r\n\t\t\t\r\n\t\t\t<input \r\n\t\t\t\ttype=\"email\" \r\n\t\t\t\t \r\n\t\t\t\tdefaultValue=\"\" \r\n\t\t\t\tclassName=\"mf-input \" \r\n\t\t\t\tid=\"mf-input-email-5a26283\" \r\n\t\t\t\tname=\"mf-email\" \r\n\t\t\t\tplaceholder=\"${ parent.decodeEntities(`Email ID`) } \" \r\n\t\t\t\t \r\n\t\t\t\tonBlur=${parent.handleChange} onFocus=${parent.handleChange} aria-invalid=${validation.errors['mf-email'] ? 'true' : 'false' } \r\n\t\t\t\tref=${el=> parent.activateValidation({\"message\":\"This field is required.\",\"emailMessage\":\"Please enter a valid Email address\",\"minLength\":1,\"maxLength\":\"\",\"type\":\"none\",\"required\":false,\"expression\":\"null\"}, el)}\r\n\t\t\t\t\t\t\t\/>\r\n\r\n\t\t\t\t\t\t<${validation.ErrorMessage} \r\n\t\t\t\terrors=${validation.errors} \r\n\t\t\t\tname=\"mf-email\" \r\n\t\t\t\tas=${html`<span className=\"mf-error-message\"><\/span>`}\r\n\t\t\t\/>\r\n\t\t\t\r\n\t\t\t\t\t<\/div>\r\n\r\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div className=\"elementor-element elementor-element-a015ee2 elementor-widget__width-auto mf-btn--center elementor-widget elementor-widget-mf-button\" data-id=\"a015ee2\" data-element_type=\"widget\" data-widget_type=\"mf-button.default\">\n\t\t\t\t<div className=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div className=\"mf-btn-wraper \" data-mf-form-conditional-logic-requirement=\"\">\r\n\t\t\t\t\t\t\t<button type=\"submit\" className=\"metform-btn metform-submit-btn \" id=\"\">\r\n\t\t\t\t\t<span>${ parent.decodeEntities(`Subscribe`) } <\/span>\r\n\t\t\t\t<\/button>\r\n\t\t\t        <\/div>\r\n        \t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t`}\r\n\t\t\t\t\t<\/div>\r\n\r\n\t\t\t\t\t${is_dummy_markup ? message_position === 'bottom' ? props.ResponseDummyMarkup(message_successIcon, message_proClass) : '' : ''}\r\n\t\t\t\t\t${is_dummy_markup ? ' ' : message_position === 'bottom' ? props.SubmitResponseMarkup`${parent}${state}${message_successIcon}${message_errorIcon}${message_proClass}` : ''}\r\n\t\t\t\t\r\n\t\t\t\t<\/form>\r\n\t\t\t`\r\n\t\t<\/script>\r\n\r\n\t\t<\/div><\/p>\n<hr \/>\n<p><a href=\"https:\/\/marketist.co\/\"><img fetchpriority=\"high\" decoding=\"async\" class=\"aligncenter wp-image-21683 size-full\" src=\"https:\/\/marketist.co\/blog\/wp-content\/uploads\/2025\/04\/Web-Design-Services-Digital-marketing-services-seo-services-google-ads-services.png\" alt=\"Web Design Services - Digital marketing services - seo services - google ads services\" width=\"750\" height=\"200\" srcset=\"https:\/\/marketist.co\/blog\/wp-content\/uploads\/2025\/04\/Web-Design-Services-Digital-marketing-services-seo-services-google-ads-services.png 750w, https:\/\/marketist.co\/blog\/wp-content\/uploads\/2025\/04\/Web-Design-Services-Digital-marketing-services-seo-services-google-ads-services-300x80.png 300w\" sizes=\"(max-width: 750px) 100vw, 750px\" \/><\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Adding a robots.txt file to a static website is a straightforward process. This file helps guide search engine crawlers on which parts of your site should be indexed. Here\u2019s how you can create and add one to your PHP-based static website. Step 1: Create the robots.txt File Create a new file named robots.txt in the [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":21557,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"default","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"set","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[4],"tags":[],"class_list":["post-21553","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-website"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.6 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>How to Add a robots.txt File to a Custom Website in PHP?<\/title>\n<meta name=\"description\" content=\"Worried adding robots.txt file to custom developed web? Here is How to Add a robots.txt File to a Custom Website in PHP. Check;\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/marketist.co\/blog\/how-to-add-a-robots-txt-file-to-a-custom-website-in-php\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to Add a robots.txt File to a Custom Website in PHP?\" \/>\n<meta property=\"og:description\" content=\"Worried adding robots.txt file to custom developed web? Here is How to Add a robots.txt File to a Custom Website in PHP. Check;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/marketist.co\/blog\/how-to-add-a-robots-txt-file-to-a-custom-website-in-php\/\" \/>\n<meta property=\"og:site_name\" content=\"Digital Marketing Blog | Marketist\" \/>\n<meta property=\"article:published_time\" content=\"2024-07-10T11:18:20+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-08-30T04:19:52+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/marketist.co\/blog\/wp-content\/uploads\/2024\/07\/robots-txt-example.png\" \/>\n\t<meta property=\"og:image:width\" content=\"464\" \/>\n\t<meta property=\"og:image:height\" content=\"292\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Marketist\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Marketist\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/marketist.co\/blog\/how-to-add-a-robots-txt-file-to-a-custom-website-in-php\/\",\"url\":\"https:\/\/marketist.co\/blog\/how-to-add-a-robots-txt-file-to-a-custom-website-in-php\/\",\"name\":\"How to Add a robots.txt File to a Custom Website in PHP?\",\"isPartOf\":{\"@id\":\"https:\/\/marketist.co\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/marketist.co\/blog\/how-to-add-a-robots-txt-file-to-a-custom-website-in-php\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/marketist.co\/blog\/how-to-add-a-robots-txt-file-to-a-custom-website-in-php\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/marketist.co\/blog\/wp-content\/uploads\/2024\/07\/robots-txt-example.png\",\"datePublished\":\"2024-07-10T11:18:20+00:00\",\"dateModified\":\"2025-08-30T04:19:52+00:00\",\"author\":{\"@id\":\"https:\/\/marketist.co\/blog\/#\/schema\/person\/74d19c0a82c40a0d070b4c40f242d7e6\"},\"description\":\"Worried adding robots.txt file to custom developed web? Here is How to Add a robots.txt File to a Custom Website in PHP. Check;\",\"breadcrumb\":{\"@id\":\"https:\/\/marketist.co\/blog\/how-to-add-a-robots-txt-file-to-a-custom-website-in-php\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/marketist.co\/blog\/how-to-add-a-robots-txt-file-to-a-custom-website-in-php\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/marketist.co\/blog\/how-to-add-a-robots-txt-file-to-a-custom-website-in-php\/#primaryimage\",\"url\":\"https:\/\/marketist.co\/blog\/wp-content\/uploads\/2024\/07\/robots-txt-example.png\",\"contentUrl\":\"https:\/\/marketist.co\/blog\/wp-content\/uploads\/2024\/07\/robots-txt-example.png\",\"width\":464,\"height\":292,\"caption\":\"How to Add a robots.txt File to a Static Custom Website in PHP\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/marketist.co\/blog\/how-to-add-a-robots-txt-file-to-a-custom-website-in-php\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/marketist.co\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How to Add a robots.txt File to a Custom Website in PHP\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/marketist.co\/blog\/#website\",\"url\":\"https:\/\/marketist.co\/blog\/\",\"name\":\"Digital Marketing Blog | Marketist\",\"description\":\"Digital Marketing Blog\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/marketist.co\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/marketist.co\/blog\/#\/schema\/person\/74d19c0a82c40a0d070b4c40f242d7e6\",\"name\":\"Marketist\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/marketist.co\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/ed65c48c415df710b496c5aa073ca625f455ed8de1ee6b5ec5702fb8ec6f9881?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/ed65c48c415df710b496c5aa073ca625f455ed8de1ee6b5ec5702fb8ec6f9881?s=96&d=mm&r=g\",\"caption\":\"Marketist\"},\"url\":\"https:\/\/marketist.co\/blog\/author\/teammarketist\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How to Add a robots.txt File to a Custom Website in PHP?","description":"Worried adding robots.txt file to custom developed web? Here is How to Add a robots.txt File to a Custom Website in PHP. Check;","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/marketist.co\/blog\/how-to-add-a-robots-txt-file-to-a-custom-website-in-php\/","og_locale":"en_US","og_type":"article","og_title":"How to Add a robots.txt File to a Custom Website in PHP?","og_description":"Worried adding robots.txt file to custom developed web? Here is How to Add a robots.txt File to a Custom Website in PHP. Check;","og_url":"https:\/\/marketist.co\/blog\/how-to-add-a-robots-txt-file-to-a-custom-website-in-php\/","og_site_name":"Digital Marketing Blog | Marketist","article_published_time":"2024-07-10T11:18:20+00:00","article_modified_time":"2025-08-30T04:19:52+00:00","og_image":[{"width":464,"height":292,"url":"https:\/\/marketist.co\/blog\/wp-content\/uploads\/2024\/07\/robots-txt-example.png","type":"image\/png"}],"author":"Marketist","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Marketist","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/marketist.co\/blog\/how-to-add-a-robots-txt-file-to-a-custom-website-in-php\/","url":"https:\/\/marketist.co\/blog\/how-to-add-a-robots-txt-file-to-a-custom-website-in-php\/","name":"How to Add a robots.txt File to a Custom Website in PHP?","isPartOf":{"@id":"https:\/\/marketist.co\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/marketist.co\/blog\/how-to-add-a-robots-txt-file-to-a-custom-website-in-php\/#primaryimage"},"image":{"@id":"https:\/\/marketist.co\/blog\/how-to-add-a-robots-txt-file-to-a-custom-website-in-php\/#primaryimage"},"thumbnailUrl":"https:\/\/marketist.co\/blog\/wp-content\/uploads\/2024\/07\/robots-txt-example.png","datePublished":"2024-07-10T11:18:20+00:00","dateModified":"2025-08-30T04:19:52+00:00","author":{"@id":"https:\/\/marketist.co\/blog\/#\/schema\/person\/74d19c0a82c40a0d070b4c40f242d7e6"},"description":"Worried adding robots.txt file to custom developed web? Here is How to Add a robots.txt File to a Custom Website in PHP. Check;","breadcrumb":{"@id":"https:\/\/marketist.co\/blog\/how-to-add-a-robots-txt-file-to-a-custom-website-in-php\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/marketist.co\/blog\/how-to-add-a-robots-txt-file-to-a-custom-website-in-php\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/marketist.co\/blog\/how-to-add-a-robots-txt-file-to-a-custom-website-in-php\/#primaryimage","url":"https:\/\/marketist.co\/blog\/wp-content\/uploads\/2024\/07\/robots-txt-example.png","contentUrl":"https:\/\/marketist.co\/blog\/wp-content\/uploads\/2024\/07\/robots-txt-example.png","width":464,"height":292,"caption":"How to Add a robots.txt File to a Static Custom Website in PHP"},{"@type":"BreadcrumbList","@id":"https:\/\/marketist.co\/blog\/how-to-add-a-robots-txt-file-to-a-custom-website-in-php\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/marketist.co\/blog\/"},{"@type":"ListItem","position":2,"name":"How to Add a robots.txt File to a Custom Website in PHP"}]},{"@type":"WebSite","@id":"https:\/\/marketist.co\/blog\/#website","url":"https:\/\/marketist.co\/blog\/","name":"Digital Marketing Blog | Marketist","description":"Digital Marketing Blog","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/marketist.co\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/marketist.co\/blog\/#\/schema\/person\/74d19c0a82c40a0d070b4c40f242d7e6","name":"Marketist","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/marketist.co\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/ed65c48c415df710b496c5aa073ca625f455ed8de1ee6b5ec5702fb8ec6f9881?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/ed65c48c415df710b496c5aa073ca625f455ed8de1ee6b5ec5702fb8ec6f9881?s=96&d=mm&r=g","caption":"Marketist"},"url":"https:\/\/marketist.co\/blog\/author\/teammarketist\/"}]}},"_links":{"self":[{"href":"https:\/\/marketist.co\/blog\/wp-json\/wp\/v2\/posts\/21553","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/marketist.co\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/marketist.co\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/marketist.co\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/marketist.co\/blog\/wp-json\/wp\/v2\/comments?post=21553"}],"version-history":[{"count":5,"href":"https:\/\/marketist.co\/blog\/wp-json\/wp\/v2\/posts\/21553\/revisions"}],"predecessor-version":[{"id":21860,"href":"https:\/\/marketist.co\/blog\/wp-json\/wp\/v2\/posts\/21553\/revisions\/21860"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/marketist.co\/blog\/wp-json\/wp\/v2\/media\/21557"}],"wp:attachment":[{"href":"https:\/\/marketist.co\/blog\/wp-json\/wp\/v2\/media?parent=21553"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/marketist.co\/blog\/wp-json\/wp\/v2\/categories?post=21553"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/marketist.co\/blog\/wp-json\/wp\/v2\/tags?post=21553"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}